Skip to content

Make precompute execution preserve post-ASAP DAG semantics end to end #762

Description

@zzylol

Problem

The precompute engine is not yet consistently driven by ASAPPlanner's post-ASAP executable DAG. The repository has a maintenance DAG scheduler, but raw-sample workers still construct accumulators from the backend's independent AggregationConfig vocabulary. A DAG in the installed plan therefore does not, by itself, establish that the runtime executes its semantics.

Concrete inconsistencies:

  • Planner Rate and Increase can become the same Increase accumulator; the payload reports Increase even for a Rate producer. Count and Sum have a similar identity problem.
  • MultipleSum, MultipleIncrease, MultipleMin, and MultipleMax mix grouping/layout into semantic operator identity, although grouping is already represented by Planner IR.
  • Unknown accumulator configurations can silently select Sum.
  • Raw ingestion executes config-selected kernels before the maintenance DAG. It must be bound to and validated against the actual selected producer node and its input path, rather than reconstructing a separate semantic program.
  • Catalog/SDS, serialized state, merge, and query readout must agree with that same Planner family. Merely copying the family into metadata is insufficient.

Required behavior

Replace the flat AggregationConfig execution format (aggregation_type, subtype, untyped parameters, and independent update rules) with a DAG installation format compatible with ASAPPlanner post-ASAP IR. Preserve Planner node IDs, edges, operator payloads, family parameters, update expressions, reductions, and readout semantics. Backend configuration should attach physical window/storage/schedule bindings to those nodes; it must not remain a second semantic program or require a handwritten MultipleX algebra. Merely wrapping the existing config or renaming accumulator modules does not satisfy this issue.

Use the selected post-ASAP DAG as the semantic source of truth. Precompute and query execution are physical projections of that DAG, split at explicit stored-summary bindings. Backend kernels may share implementation code, but must preserve distinct Planner family identities and reject invalid reads/merges.

Acceptance

  • Raw precompute kernels are installed from validated Planner producer nodes and dependency paths; unsupported or inconsistent paths fail explicitly before execution.
  • Existing maintenance DAG execution remains wired into production, retaining dependency ordering and shared-node execution.
  • Runtime exact states preserve Sum, Count, Min, Max, Rate, and Increase across update, merge, serialization, storage, and readout. Grouped layout is not an alternate MultipleX semantic family.
  • Unknown or unsupported configurations cannot fall back to Sum.
  • Catalog/SDS and both physical projections validate the same Planner family and node bindings.
  • Regression tests demonstrate the old failures and cover valid execution plus wrong-family rejection. Integration coverage exercises installed DAG -> ingestion -> stored state -> query readout, including counter reset and Rate versus Increase.

Implement in a dedicated prerequisite PR based on main. Stack #728 on it, followed by #742 and #759. Keep issue #754's query-specific plan acceptance and differential/benefit suites in those testing PRs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions