Skip to content

refactor: execute query DAGs with shared native physical operators - #765

Open
zzylol wants to merge 48 commits into
fix/precompute-post-asap-dagfrom
refactor/query-plan-dag-execution
Open

zzylol wants to merge 48 commits into
fix/precompute-post-asap-dagfrom
refactor/query-plan-dag-execution

Conversation

@zzylol

@zzylol zzylol commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Dependency stack: main → #768 → #737 → #749 → #771 → #728 → #770 → #763 → #765 → #761 → #742 → #759

Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).

Closes #764. Stacked on #763; the Planner-owned library is consumed through #770.

Before this PR

Installed query execution mixed backend traversal, duplicated relation/temporal computations and specialized candidate/ranking representations. Classifying a physical operator did not prove that its schema and parameters could execute.

After this PR

Both engines use the independent DAG runtime from Planner #462. SQL relations bind through the shared post-ASAP binder and execute in one native DAG. PromQL bindings send values and windows to shared Project, aggregation, semi-join, grouped Sort/Limit, temporal and histogram operators. Binary computation uses Planner kinds and checked-division contracts.

Stored-summary codecs, delta reconstruction, merge and readout computation now live in Planner. The backend selects sources/panes, validates population and window coverage, and converts protocol values. It does not carry a second copy of those algorithms. Relation installation rejects unsupported expressions/types through the same binder used for execution; source conversion preserves the complete Planner schema, including its time index.

SQL storage frontiers are evaluated together in one DAG run, retaining pane coverage and sharing the same parent cancellation signal and resource budget. A producer shared by two consumers executes once per run. Native calls preserve the caller's cancellation and resource budget. SQL empty/missing window inputs are classified as incomplete coverage. Raw local Scan remains deferred.

Candidate pruning is a general semi-join with explicit keys and completeness evidence, followed by Sort → Limit within each group. There is no dedicated MembershipFilter or grouped TopK physical operator.

Design and acceptance contract. The table below distinguishes implemented shared computation from remaining deployment capabilities; ingestion time and query time are phases, not separate operator definitions.

Physical operation coverage

Physical operation Purpose Shared library implementation Backend integration Missing coverage
Raw Scan Read raw input rows Engines can supply typed batch sources Local raw Scan is rejected Local raw source is explicitly deferred
Read materialization Load previously computed state Shared decoding and delta reconstruction Catalog/store references, population and window checks Missing or incompatible state is a runtime error
Maintain current-series state Update values and timestamps of maintained time series No general table update operation Remote-write storage path General row updates; this is not a raw-table executor
Read current-series state Read maintained current values Typed source interface Installed identity/capacity and time scope Arbitrary raw-table reading
Scalar Produce a scalar Typed source, including nullable values Query scalar source General ingestion literal binding
Binary Combine or compare values Planner numeric/comparison kinds, typed expressions and checked-division domains Query native Project; ingestion uses shared arithmetic kernels Arbitrary vector matching and unsupported domains
Unary negate Negate a value Int64/Float64 expression, null propagation and checked integer overflow Query native Project General ingestion expression binding
Vector to scalar Convert an instant vector to a scalar Native Float64 operator; zero or multiple rows yield NaN Query native batch binding General ingestion value binding
Exact aggregate Compute Count/Sum/Avg/Min/Max, including ReduceSum Native grouped aggregation; checked Int64 Sum, Float64 Sum/Avg, Int64 Count, ordered Min/Max and nullable inputs Query vector and SQL relation bindings use shared computation Other aggregate intents, unresolved grouping-without and per-entity relation lowering; no spill
Finalize exact accumulator / ExactReadout Obtain an exact result from state Shared state merge/finalization and typed batch readout Ingestion finalization and stored-query readout use shared kernels Arbitrary state conversion and unsupported exact families
Project Select or compute output columns Planner scalar expressions and native projection SQL native DAG; vector expression binding Unsupported expression functions/types
Filter Keep rows satisfying a predicate Planner predicates, three-valued boolean logic and native filtering SQL native DAG Unsupported predicates/coercions
Relational join, including semi-join Match rows; semi-join retains matching left rows Inner/left/right/full/cross/semi/anti joins; typed predicates and nullable outer outputs SQL native DAG; candidate vectors use general semi-join with explicit keys General ingestion source binding; SQL candidate-pruning source binding
Sort Order rows or values Stable grouped sort with null placement Query vector and SQL native Sort Sort expressions must be projected first; unsupported key types; no spill
Limit Keep a bounded slice per group Grouped offset/limit across batches Planner grouped Limit; query vector and SQL native binding Limit does not rank or match candidate keys
Union Combine compatible input streams Fair polling of native input streams Multi-input summary merge General installed batch-source binding
SummaryAgg Construct summary state Native builders for exact Sum/Count/Min/Max/Rate/Increase, KLL, DDSketch and HLL; other admitted families have shared update kernels Ingestion DAG and per-window update paths General query builder binding; native Int64/keyed updates and other batch families
SummaryMerge Combine compatible summary states Native grouped merge and shared stored-state merge kernels Ingestion native DAG; query selects panes and invokes shared state kernels Cross-family conversion is not a merge; not every stored encoding has a native batch binding
SummaryEstimate Read an approximate result Shared Planner SketchQuery dispatch; native KLL/DDSketch/HLL batch readout Stored-state query readout uses shared kernels Unsupported family/readout combinations; accuracy evidence remains required
SummaryJoin Combine summaries using summary join semantics No registered implementation Rejected Concrete semantics and implementation
SummarySubtract Subtract summary state No registered implementation Rejected Concrete semantics and implementation
SummaryDelete Remove state contributions No registered implementation Rejected Concrete semantics and implementation
Temporal computation Compute Rate/Increase/Sum/Avg/Min/Max/Count over a window Native window operator using Planner intents; reset-aware rates, Int64 Count Query range vectors bind to native batches Other temporal intents; general ingestion window binding
Histogram quantile Interpolate a quantile from histogram buckets Native window operator using Planner HistogramQuantile intent Query bucket input binds to native batches General ingestion histogram source binding
Subquery Evaluate an expression over a time grid Shared DAG execution and request-local caching of intermediate results Backend binds a bounded grid of operation/evaluation-time nodes General ingestion time-grid binding
Extension Execute an additional operation No universal executor Unsupported extensions are rejected A registered implementation for each admitted extension

Validation

Planner: 246 shared-library unit tests, 18 library integration tests, one doc test, 432 mapping tests and the integration package passed. Both repositories pass strict workspace/all-target Clippy.

Backend stack: 118 type and 444 control-plane library tests passed. The data-plane run passed 925 tests; its remaining persistence synchronization test was fixed and passed separately. All 19 compatibility process tests and 59 other integration tests passed. Tests cover shared SQL sources, cancellation, stored-state coverage, temporal Sort/Limit, HLL/KLL confidence, and ingestion publication/recovery.

The mandatory #754 gate still fails on grouped-temporal-Sum producer shape and quantile-ratio local execution. Those assertions remain intact. Full #759 performance acceptance is not established. Local raw Scan remains explicitly deferred.

The publication process test verifies that a successor is cold before its own data arrives, rejects old-generation frames and returns the successor's new values; it does not assume cross-version payload reuse.

@zzylol zzylol changed the title refactor: execute installed QueryPlan sub-DAGs refactor: share physical operator kernels and execute QueryPlan DAGs Sep 23, 2026
@zzylol zzylol changed the title refactor: share physical operator kernels and execute QueryPlan DAGs refactor: execute query DAGs with shared native physical operators Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant