Skip to content

feat(control-plane): scope evidence and compute ERP/analytical workload costs - #761

Open
zzylol wants to merge 1 commit into
refactor/query-plan-dag-executionfrom
issue-752
Open

zzylol wants to merge 1 commit into
refactor/query-plan-dag-executionfrom
issue-752

Conversation

@zzylol

@zzylol zzylol commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Dependency stack: main → #768 → #737 → #749 → #771 → #728 → #770 → #763 → #765 → #761 → #742 → #759

Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).

Stacked directly on #765. Scoped evidence and workload costing depend on installed query execution, not diagnostic logging, the differential runner or overhead inspection. Closes #752.

Before this PR

The backend can select costed Planner candidates, but query/data/snapshot-scoped external evidence and ERP resource measurements are not integrated into complete backend-owned workload costing. Callers must provide complete physical-candidate quotes.

After this PR

Scoped evidence supplies quantile-operand domains, Hydra bounds, TopK intervals and distribution/cardinality facts, with scope/freshness validation. Applicable ERP benchmark resources take priority over explicitly identified analytical estimates; resource measurements do not certify accuracy. Both startup and compile/publish combine unit resources with query recurrence, data scale, shared state, physical layouts and transport, calculate complete workload costs and compare feasible physical candidates. Provider quotes remain an explicit optional override.

Source cadence, query lookback, ranges and offsets retain millisecond precision. A real 100 ms workload now reaches compilation/cost comparison without rounding the source cadence. Existing temporal storage-family capability checks still apply; this does not claim every pane implementation supports fractional windows.

Design: docs/design_docs/evidence-dependent-candidates.md. The Planner dependency inherited from #768 includes ProjectASAP/ASAPPlanner#460. Scoped HLL evidence can opt into its classic 64-bit linear-counting confidence bound for complete readout populations of at most 4,096 distinct values, under the independent uniform bucket-hash assumption. The contract covers merged panes and is restricted to the verified local Remote Write estimator. Missing, infeasible or unsupported contracts retain exact execution; ERP resource costs do not substitute for probability guarantees.

Verification

  • 443 control-plane library tests pass, including ERP precedence, automatic workload costing, scoped evidence and millisecond-history/100 ms publication regressions.
  • 8 control-plane binary tests and discovery integration pass.
  • test: measure issue 754 level-3 benefit against exact DBs #759's 12 Rust runner tests pass; the real 100 ms fixture now compiles and validates automatic cost coverage.
  • cargo clippy --workspace --all-targets --locked -- -D warnings, cargo fmt --all -- --check, and remaining exact-fallback/offline-evidence integrations pass.

Acceptance boundary

#759 remains stacked directly on this PR and is the full level-3 acceptance layer. make benefit now passes workload compilation/cost validation, then fails the strict local-execution gate because compat-query-9 requires external exact execution. The inherited level-1 test still reports grouped temporal Sum and quantile-ratio plan-shape failures. No full measured-benefit success is claimed, and these gates have not been relaxed.

Final stacked process validation: all 19 asapquery_compatibility_process_e2e tests pass, including confidence-sized KLL state transport, bounded classic-HLL execution, and exact routing for uncertified empirical evidence. The KLL regression also passes independently on #749.

Final #759 stack checks: all 12 Rust runner tests pass, as do cargo clippy --workspace --all-targets --locked -- -D warnings, cargo fmt --all -- --check and git diff --check.

Shared physical operator stack update

The shared runtime, operators and sketch codec are owned by ASAPPlanner #462, built on Planner #461. Backend #770 consumes them, #763 integrates ingestion and #765 integrates queries. The stack uses general semi-join and grouped Sort → Limit, with execution phase on the node. Independent library tests run in Planner; deployment acceptance stays in this backend stack.

Automatic workload costing prices general semi-join/anti-join output cardinality and the supported equality semi-join implementation; no dedicated membership operator remains.

@zzylol
zzylol changed the base branch from main to refactor/query-plan-dag-execution September 22, 2026 20:20
@zzylol
zzylol changed the base branch from refactor/query-plan-dag-execution to perf/issue-758 September 22, 2026 20:45
@zzylol zzylol changed the title feat(control-plane): gate evidence-dependent Planner candidates feat(control-plane): scope evidence and compute ERP/analytical workload costs Sep 23, 2026
@zzylol
zzylol changed the base branch from perf/issue-758 to refactor/query-plan-dag-execution September 24, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate Planner evidence-dependent candidates into backend selection

1 participant