feat: share physical operators and DAG execution across precompute and query deployments

[zzylol]

Problem

Precompute and query deployments need to execute Planner-selected computations with consistent semantics. Previously, physical computation lived in ASAPQuery-backend and depended on backend types; there was no independent physical-operator library and DAG runtime for deployments to share. Planner IR and deployment implementations could evolve separately.

Before this PR

Deployments could not import a shared Planner-owned implementation to bind and execute a selected physical DAG. Supporting a new IR operation required coordinating Planner semantics with backend-specific execution code.

After this PR

This PR provides both a physical-operator library and a DAG execution runtime in asap-physical-operators:

• Native relational, temporal and summary operators, including summary build/merge/readout and weighted CMS/CountSketch TopK.
• Binding from Planner's executable DAG to concrete physical operators.
• Shared-producer execution, independent per-run state, schema validation, backpressure, cooperative cancellation and estimated memory accounting.

The plan-to-execution path is:

SQL / PromQL → QueryExpr → ASAP planning / summary selection → SummaryExpr
    → compile_executable_dag(...) → ExecutableDag
    → binding::bind(...) + deployment inputs → PhysicalDag
    → execute(..., RunContext) → results

ExecutableDag describes the selected operations, schemas and dependencies. PhysicalDag binds that description to concrete operator implementations. Deployments provide raw-source connectors or stored-state frontiers and select the DAG/subDAG roots to execute.

For example, both deployment phases can use the same rate → weighted summary → candidate readout → grouped Sort/Limit computation. The intended result is unchanged; the change is a common, independently testable implementation and execution path. Keeping IR and physical execution in ASAPPlanner lets a new operation be developed in one PR and tested across the planning/execution boundary, including private implementation APIs.

Weighted CMS/CountSketch matrices, hashing, estimation, candidate heaps and snapshots are owned by asap_sketchlib; summary_kernels/weighted_frequency.rs now only adapts Planner parameters, typed values and runtime traits. This PR pins sketchlib commit 5f03ccbd798ed5fec62bdd839bcb331123cab369, supplied by asap_sketchlib #160 (pending merge). The existing ASAP-WFREQ-1 snapshots remain compatible. The dependency upgrade also preserves DDSketch negative/zero stores in full-state and delta adapters.

Deployment responsibilities

asap-fusion and ASAPQuery remain responsible for their precompute/query engine orchestration: source access, scheduling runs, evaluation windows and revisions, persistent state, publication and serving. They import the shared operators and DAG runtime to execute the selected DAG/subDAG. Deployment-specific runtimes wrap this execution infrastructure.

This PR establishes the reusable library path; adopting it in deployed engines and verifying real storage-to-serving workflows remain backend integration work.

E2E behavior and current evidence

Test level	Expected behavior	Existing coverage
Individual operators	Correct results, supported types, edge cases, errors and resource constraints	Unit tests and operator contract tests; coverage is not an exhaustive cross-product of every operator and edge case
Physical DAG execution	Correct composition, shared producers, dependency handling, cancellation and independent runs	DAG integration tests, including summary build/merge/multiple readouts, shared branches and resource failures
Planner-generated DAG binding and execution	Selected Planner operations bind to the corresponding implementations and produce expected results	PromQL weighted TopK runs through planning, DAG compilation, binding and execution in both phases, with input supplied at the rate-value frontier

The raw Scan → Sort → Limit tests use a manually constructed executable DAG. The Planner TopK test supplies already-computed rate values. Full SQL/PromQL text → raw in-memory input → Planner → native execution coverage still needs to be added. SQL-to-post-ASAP tests establish planning coverage, not full native execution.

Automated tests were run locally. No separate manual deployment-level verification has been performed.

• Physical library: 252 unit tests, 50 integration tests and one documentation test passed.
• Planner mapping: 438 unit tests; types: 211 unit tests; relevant exact-composition/PromQL/SQL integration: 54 tests passed.
• Strict workspace Clippy, formatting and whitespace checks passed.
• Six semantic/resource regressions were reproduced before fixes, covering global extrema, NaN comparisons and joins, expression binding and row memory accounting. The added contract tests borrow DataFusion test ideas without introducing a DataFusion execution dependency.

Scope

ASAP implements and maintains the operators and runtime, using DataFusion's organization and execution contracts as a reference. Parallelism, sharding, spill and richer physical optimization are future capabilities that can be implemented in this shared library. They are not provided by this PR, and no performance improvement or complete SQL semantic equivalence is claimed.

Architecture and operator coverage · DataFusion comparison

[milindsrivastava1997]

High-level questions @zzylol:

• What is the aim of this PR? Is it to create a separate library of physical operators? Is it to create a runtime for those physical operators? Both?
• What are the E2E behavior to be tested here? Are there tests for this? Have these been verified manually?
• What is the before vs after E2E effect of this PR?
• Given this PR, what is the expected role of asap-fusion and ASAPQuery?

[zzylol]

High-level questions @zzylol:

• What is the aim of this PR? Is it to create a separate library of physical operators? Is it to create a runtime for those physical operators? Both?

Both.

• What are the E2E behavior to be tested here? Are there tests for this? Have these been verified manually?

There are two levels:

1. Each physical operator produces correct results and handles its supported types, edge cases, errors and resource constraints.
2. Given a physical DAG, the operators execute together correctly, including dependencies, shared producers and independent execution runs.

Both have automated coverage: operator tests and DAG integration tests. These were run locally. No separate manual deployment-level verification has been performed.

The physical DAG (https://github.com/ProjectASAP/ASAPPlanner/blob/2216fb9/crates/asap-physical-operators/src/plan/mod.rs) comes from the Planner’s post-ASAP plan, followed by binding to concrete operators:

SQL / PromQL
↓ frontend
QueryExpr
↓ ASAP planning and summary selection
SummaryExpr
↓ compile_executable_dag(...)
ExecutableDag
↓ binding::bind(...) + deployment sources
PhysicalDag
↓ execute(..., RunContext)
Results

There are two distinct DAGs:

• ExecutableDag describes the selected computation, schemas and shared dependencies. Despite its name, it does not contain running physical operators.
• PhysicalDag contains concrete implementations from the shared library. The binder constructs it from ExecutableDag; deployments provide sources or stored-state frontiers and select execution roots.

So deployments reuse Planner-generated plans plus the shared binding/execution infrastructure. They do not need to construct physical DAGs manually, although low-level tests can.

This also adds a third testing requirement: verify that Planner-generated DAGs bind correctly, beyond testing individual operators and manually assembled DAGs.

• What is the before vs after E2E effect of this PR?

Before: There was no shared physical-operator library for deployments to reuse.

After: Deployments can reuse the same physical-operator implementations and physical DAG execution infrastructure. This also provides a common place to implement future physical execution optimizations, such as parallelism and sharding. Those optimizations are not implemented in this PR.

• Given this PR, what is the expected role of asap-fusion and ASAPQuery?

asap-fusion and asapquery will be the deployment, which refer/import the physical operator lib here, and implement their own runtimes, e.g., the precompute engine with a DAG / subDAG, query engine. Putting the phsycial opereator lib together with ASAPPlanner is as Yancheng said the benefits of

If we consider organizing the physical operator library inside the ASAPPlanner repo, it has two benefits:

when a new IR node is added, we just need one PR for changing both the logical DAG and the physical operator implementation.
It's easier for testing because some APIs are private and it can broaden the testing coverage because logical ASAP IR and physical operators should be consistent.

[milindsrivastava1997]

• What does "Both have automated coverage: " mean?
• What are e2e correctness tests we can add here?
• Did not understand Executable vs Physical DAG. Please give an example and explain why we need two of these
• "deployments provide sources or stored-state frontiers and select execution roots." -- did not understand
• What is the difference between a deployment and a runtime?
• Can we please rename binding? I have spent many hours being confused by this term in earlier docs too
• For each physical operator defined here, is there a description of its behavior? Property based testing might be useful here
• For each physical operator here, is there a clear mapping to a post-ASAP IR node?
• We definitely need to split this PR. I would suggest first only adding a library of physical operators. Then modify ASAPQuery-backend and asap-fusion to use this library?

[milindsrivastava1997]

what is a summary kernel

[zzylol]

summary_kernels/ sketchlib adapters, exact accumulators, factory and traits