Conversation
This was referenced Sep 22, 2026
Integrate PR #749 reader/writer bindings and maintenance projections while preserving data-partition worker ownership and plan-derived configuration. Keep selected and maintenance DAG schemas distinct and adapt shared-sink execution to the projected graph.
zzylol
changed the base branch from
docs/physical-plan-design
to
refactor/backend-plan-split
September 22, 2026 19:35
Carry StoredOutputReference through plan installation, worker output, publication, query lookup, and durable metadata. Key output populations by plan identity/version and selected output, retaining local numeric handles only for physical row lookup. Reject implicit cross-version reads and old metadata formats; preserve complete-source publication barriers. Verify independent output IDs, restart recovery, and relocation-independent maintenance lineage.
This was referenced Sep 22, 2026
zzylol
changed the base branch from
refactor/backend-plan-split
to
test/promql-exact-function-coverage
September 22, 2026 20:45
This was referenced Sep 22, 2026
zzylol
changed the base branch from
test/promql-exact-function-coverage
to
feat/shared-operator-foundation
September 23, 2026 19:27
…ompute-763 # Conflicts: # data_plane/src/precompute_engine/maintenance_runtime.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dependency stack: main → #768 → #737 → #749 → #771 → #728 → #770 → #763 → #765 → #761 → #742 → #759
Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).
Related to #762. Stacked directly on #770, the Planner-library consumption foundation.
Before this PR: the runtime could execute a flat aggregation configuration independently of Planner's selected DAG, represent Rate using an Increase family, and assign workers using storage SIDs. The storage path in #749 bound reads through a definition-keyed SID index and required the stored-output ID to equal its definition ID.
After this PR: execution is derived from the installed physical plan, and stored state is bound to the selected DAG output. For example, a Rate producer keeps its Planner family; output
101can refer to a different numeric definition ID, and its writer and query readers agree on the same output, plan version, population and window after a process restart.StreamingConfigloading, CLI and HTTP configuration withInstalledPrecomputePlanderived from the executable DAG. Remove backendMultiple*families and preserve Planner operator/update/reduction semantics.PhysicalDagruntime and preserve binary input ordering. Multiple sinks share upstream producers within one run. Completed-window SummaryAgg, SummaryMerge, exact finalization and aligned Binary use native library operators; the backend supplies batch conversion, completeness checks and publication. Native calls retain the parent memory budget and cancellation scope.StoredOutputReferencebindings with the plan. Publish usingStoredSummaryKey(plan_id, plan_version, output, population, window); bound queries select the exact output rather than substitute another output sharing its definition.storage_handlevalues are local row locators, not logical output identities. Native ingest, backfill and maintenance allocate them from the selected output population identity. Cached handles from a different plan version or output are rejected.Precompute execution design describes inputs, partitioning, worker execution, publication and recovery with a shared KLL example.
Remaining design acceptance work: grouped derived DAG parallelism and the complete query-specific #754 matrix are not established here. This PR is not a claim that every part of the overall execution design is complete.
Shared-library integration: the Planner library is consumed through #770. Completed-window operators share the parent context and enforce Planner checked-division domains. Raw per-window updates use shared-library updaters; storage, population completeness, window routing and publication remain backend responsibilities. Query integration follows in #765.
Activation acceptance: a successor cannot read the previous plan version's payload. The process test checks the cold successor, rejects old-version frames, and verifies the successor's own new output. Atomic publication does not promise uninterrupted warm service; that requires an explicit readiness/state-migration protocol.
Validation: the production control-plane → data-plane process test passes with generation-isolated stored outputs. The 19 compatibility process tests also pass. The inherited #754 grouped-temporal-Sum and quantile-ratio gates still fail; full #759 performance acceptance is not established.