Conversation
There was a problem hiding this comment.
@zzylol
The differential workflow is currently deterministically red for two independent reasons.
-
ComposeLifecycle.Startstarts the one-shotplannerwithdocker compose up -d …, then separately callsdocker compose wait planner.plannercan successfully exit in the interval between those commands, leavingwaitwith no container to observe (no containers for project ...). This already prevents the existing corpora from running. Please start and await the planner atomically (for example, anupinvocation that waits and propagates the planner exit code), or otherwise synchronize on the generated snapshot. -
The new issue-754 corpus contains
quantile_over_time(0.9, data[1m]) / quantile_over_time(0.5, data[1m]), which the selected-plan gate explicitly classifies as requiringexact_fallback/. The subsequent benefit measurements therefore can never run, and the required CI job stays red. This is a valid semantic gate result, but it means the PR is not merge-ready as a level-3 benchmark: implement local execution for that query, or remove/mark it as unsupported until that work lands.
cf0395b to
59e848b
Compare
4c91b8d to
c338080
Compare
Dependency stack: main → #768 → #737 → #749 → #771 → #728 → #770 → #763 → #765 → #761 → #742 → #759
Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).
Stacked directly on #742. The shared Rust correctness runner and #761 automatic costing are inherited. This PR adds five benefit-specific files/changes: workflow, documentation, Compose extension, runner entry point and Make target.
Before this PR
Correctness fixtures did not measure benefit against all three exact databases, and synthetic complete quotes bypassed automatic ERP/analytical workload costing.
After this PR
A pure Rust runner builds an unquoted snapshot from actual replay cadence, series count, sample volume, aggregate arrival rate and query recurrence. It validates complete backend-computed resources, totals and provenance, then enforces local-execution and semantic gates before measuring p50/p95 latency, cgroup CPU and peak memory against Prometheus, VictoriaMetrics and ClickHouse. Remote Write, SQL generation, validation and reporting use the Cargo workspace; Go tooling is removed.
The eight-series fixture now retains 100 ms in both data demand and
implementation.scrape_interval_ms. A Rust regression compiles this real snapshot and checks automatic cost coverage. There is no rounding to one second or synthetic forced winner.Verification and limits
issue754_level1gate still reports grouped-temporal-Sum producer-shape and quantile-ratio local-execution failures. Its assertions are retained.make benefitattempt passed compilation and automatic costing at 100 ms, then stopped before Docker atcompat-query-9 requires external exact execution. Full live benefit is not established.The default fixture exercises analytical costing; it does not invent ERP measurements. ERP precedence is covered by Rust contract tests. The fixed fixture does not establish the full cardinality/time-range matrix requested by #754.
Shared physical operator stack update
The shared runtime, operators and sketch codec are owned by ASAPPlanner #462, built on Planner #461. Backend #770 consumes them, #763 integrates ingestion and #765 integrates queries. The stack uses general semi-join and grouped Sort → Limit, with execution phase on the node. Independent library tests run in Planner; deployment acceptance stays in this backend stack.
This PR remains the end-to-end workload/performance acceptance layer. Compilation or library tests alone do not establish that acceptance.
Scope verification
The reorganized #759 and independent #766 trees pass strict workspace/all-target Clippy. Shared Rust runner tests pass. This does not resolve the existing #754 plan-shape gates or establish live performance benefit.