Skip to content

refactor: execute precompute DAGs with shared physical operators - #763

Open
zzylol wants to merge 45 commits into
feat/shared-operator-foundationfrom
fix/precompute-post-asap-dag
Open

zzylol wants to merge 45 commits into
feat/shared-operator-foundationfrom
fix/precompute-post-asap-dag

Conversation

@zzylol

@zzylol zzylol commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Dependency stack: main → #768 → #737 → #749 → #771 → #728 → #770 → #763 → #765 → #761 → #742 → #759

Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).

Related to #762. Stacked directly on #770, the Planner-library consumption foundation.

Before this PR: the runtime could execute a flat aggregation configuration independently of Planner's selected DAG, represent Rate using an Increase family, and assign workers using storage SIDs. The storage path in #749 bound reads through a definition-keyed SID index and required the stored-output ID to equal its definition ID.

After this PR: execution is derived from the installed physical plan, and stored state is bound to the selected DAG output. For example, a Rate producer keeps its Planner family; output 101 can refer to a different numeric definition ID, and its writer and query readers agree on the same output, plan version, population and window after a process restart.

  • Replace production StreamingConfig loading, CLI and HTTP configuration with InstalledPrecomputePlan derived from the executable DAG. Remove backend Multiple* families and preserve Planner operator/update/reduction semantics.
  • Route each data partition to one worker. Workers own separate maintenance contexts; derived graphs conservatively use one worker until finer locality can be proved. Finite maintenance runs through worker messages after durable source completion.
  • Execute selected sinks through the shared PhysicalDag runtime and preserve binary input ordering. Multiple sinks share upstream producers within one run. Completed-window SummaryAgg, SummaryMerge, exact finalization and aligned Binary use native library operators; the backend supplies batch conversion, completeness checks and publication. Native calls retain the parent memory budget and cancellation scope.
  • Install actual StoredOutputReference bindings with the plan. Publish using StoredSummaryKey(plan_id, plan_version, output, population, window); bound queries select the exact output rather than substitute another output sharing its definition.
  • Persist output references and catalog generations. Numeric storage_handle values are local row locators, not logical output identities. Native ingest, backfill and maintenance allocate them from the selected output population identity. Cached handles from a different plan version or output are rejected.
  • Use output references, generation, population, windows and state in immutable maintenance input fingerprints; relocating physical rows does not change lineage.
  • Reject implicit cross-version payload reuse. Historical source populations still prevent certifying an incomplete subset as a complete maintenance input.
  • Use durable metadata schema 4. Earlier SID metadata is not automatically rebound or migrated; existing state must be rebuilt. Update storage metadata APIs and design documents accordingly.

Precompute execution design describes inputs, partitioning, worker execution, publication and recovery with a shared KLL example.

Remaining design acceptance work: grouped derived DAG parallelism and the complete query-specific #754 matrix are not established here. This PR is not a claim that every part of the overall execution design is complete.

Shared-library integration: the Planner library is consumed through #770. Completed-window operators share the parent context and enforce Planner checked-division domains. Raw per-window updates use shared-library updaters; storage, population completeness, window routing and publication remain backend responsibilities. Query integration follows in #765.

Activation acceptance: a successor cannot read the previous plan version's payload. The process test checks the cold successor, rejects old-version frames, and verifies the successor's own new output. Atomic publication does not promise uninterrupted warm service; that requires an explicit readiness/state-migration protocol.

Validation: the production control-plane → data-plane process test passes with generation-isolated stored outputs. The 19 compatibility process tests also pass. The inherited #754 grouped-temporal-Sum and quantile-ratio gates still fail; full #759 performance acceptance is not established.

@zzylol
zzylol changed the base branch from main to docs/physical-plan-design September 22, 2026 18:10
Integrate PR #749 reader/writer bindings and maintenance projections while preserving data-partition worker ownership and plan-derived configuration. Keep selected and maintenance DAG schemas distinct and adapt shared-sink execution to the projected graph.
@zzylol
zzylol changed the base branch from docs/physical-plan-design to refactor/backend-plan-split September 22, 2026 19:35
Carry StoredOutputReference through plan installation, worker output, publication, query lookup, and durable metadata. Key output populations by plan identity/version and selected output, retaining local numeric handles only for physical row lookup. Reject implicit cross-version reads and old metadata formats; preserve complete-source publication barriers. Verify independent output IDs, restart recovery, and relocation-independent maintenance lineage.
@zzylol zzylol changed the title refactor: execute precompute from Planner DAGs and preserve exact families refactor: execute precompute DAGs with shared physical operators Sep 23, 2026
@zzylol
zzylol changed the base branch from test/promql-exact-function-coverage to feat/shared-operator-foundation September 23, 2026 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make precompute execution preserve post-ASAP DAG semantics end to end

1 participant