Skip to content

refactor: consume Planner-owned physical DAG operators - #770

Open
zzylol wants to merge 16 commits into
test/promql-exact-function-coveragefrom
feat/shared-operator-foundation
Open

zzylol wants to merge 16 commits into
test/promql-exact-function-coveragefrom
feat/shared-operator-foundation

Conversation

@zzylol

@zzylol zzylol commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Dependency stack: main → #768 → #737 → #749 → #771 → #728 → #770 → #763 → #765 → #761 → #742 → #759

Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).

Before this PR

Shared execution and mathematical kernels lived in the backend, while Planner's IR could change independently. Candidate pruning and grouped ranking still had dedicated backend representations.

After this PR

Consume asap-physical-operators and asap_sketch_codec from Planner #462, pinned with Planner IR to revision 645cacb04a451e25147f0d23aaddd05da16b4746. Remove the backend copies. The library depends on Planner types and has no backend dependency.

The API follows Planner #461: phase belongs to node data state, candidate filtering uses a general semi-join, and grouped ranking composes Sort → grouped Limit. Old operator dispatch and wire compatibility paths are removed. Exact accumulator state is explicitly finalized before value consumers. Temporal exact TopK candidates also lower to grouped Sort → Limit, including integer Count scores.

The independent runtime owns shared producers, bounded delivery, cancellation and retained-output accounting. Native computation includes expressions, relations, window reductions and supported summary operations. The same implementation can run at ingestion time or query time. Custom summary values need not fit Arrow RecordBatch.

Backend #763 integrates ingestion and durable publication; #765 integrates query execution and completes relation/temporal/stored-state computation migration. Sources, storage and protocol conversion remain deployment responsibilities. Local raw Scan is deferred; blocking operators have no spill support.

Architecture and DataFusion comparison. Independent library tests run in Planner; backend tests exercise deployment bindings.

Validation

Planner: 246 shared-library unit tests, 18 library integration tests, one doc test, 432 mapping tests and the integration package passed. Both repositories pass strict workspace/all-target Clippy.

Backend stack: 118 type and 444 control-plane library tests passed. The data-plane run passed 925 tests; its remaining persistence synchronization test was fixed and passed separately. All 19 compatibility process tests and 59 other integration tests passed. Tests cover shared SQL sources, cancellation, stored-state coverage, temporal Sort/Limit, HLL/KLL confidence, and ingestion publication/recovery.

The mandatory #754 gate still fails on grouped-temporal-Sum producer shape and quantile-ratio local execution. Those assertions remain intact. Full #759 performance acceptance is not established. Local raw Scan remains explicitly deferred.

The publication process test verifies that a successor is cold before its own data arrives, rejects old-generation frames and returns the successor's new values; it does not assume cross-version payload reuse.

@zzylol zzylol changed the title feat: independent shared physical operator DAG library refactor: consume Planner-owned physical DAG operators Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant