Skip to content

test: measure issue 754 level-3 benefit against exact DBs - #759

Draft
zzylol wants to merge 1 commit into
732-test-add-prometheus-remote-write-promql-differential-suitefrom
test/issue754-level3
Draft

zzylol wants to merge 1 commit into
732-test-add-prometheus-remote-write-promql-differential-suitefrom
test/issue754-level3

Conversation

@zzylol

@zzylol zzylol commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Dependency stack: main → #768 → #737 → #749 → #771 → #728 → #770 → #763 → #765 → #761 → #742 → #759

Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).

Stacked directly on #742. The shared Rust correctness runner and #761 automatic costing are inherited. This PR adds five benefit-specific files/changes: workflow, documentation, Compose extension, runner entry point and Make target.

Before this PR

Correctness fixtures did not measure benefit against all three exact databases, and synthetic complete quotes bypassed automatic ERP/analytical workload costing.

After this PR

A pure Rust runner builds an unquoted snapshot from actual replay cadence, series count, sample volume, aggregate arrival rate and query recurrence. It validates complete backend-computed resources, totals and provenance, then enforces local-execution and semantic gates before measuring p50/p95 latency, cgroup CPU and peak memory against Prometheus, VictoriaMetrics and ClickHouse. Remote Write, SQL generation, validation and reporting use the Cargo workspace; Go tooling is removed.

The eight-series fixture now retains 100 ms in both data demand and implementation.scrape_interval_ms. A Rust regression compiles this real snapshot and checks automatic cost coverage. There is no rounding to one second or synthetic forced winner.

Verification and limits

  • All 19 compatibility process tests passed, including confidence-sized KLL, bounded HLL, uncertified ERP exact routing, and durable DAG maintenance.
  • Type/control-plane library suites passed (118/444 tests); 59 other integration tests passed, including the Rust runner tests. Strict workspace/all-target Clippy and formatting checks passed.
  • The mandatory issue754_level1 gate still reports grouped-temporal-Sum producer-shape and quantile-ratio local-execution failures. Its assertions are retained.
  • The last make benefit attempt passed compilation and automatic costing at 100 ms, then stopped before Docker at compat-query-9 requires external exact execution. Full live benefit is not established.

The default fixture exercises analytical costing; it does not invent ERP measurements. ERP precedence is covered by Rust contract tests. The fixed fixture does not establish the full cardinality/time-range matrix requested by #754.

Shared physical operator stack update

The shared runtime, operators and sketch codec are owned by ASAPPlanner #462, built on Planner #461. Backend #770 consumes them, #763 integrates ingestion and #765 integrates queries. The stack uses general semi-join and grouped Sort → Limit, with execution phase on the node. Independent library tests run in Planner; deployment acceptance stays in this backend stack.

This PR remains the end-to-end workload/performance acceptance layer. Compilation or library tests alone do not establish that acceptance.

Scope verification

The reorganized #759 and independent #766 trees pass strict workspace/all-target Clippy. Shared Rust runner tests pass. This does not resolve the existing #754 plan-shape gates or establish live performance benefit.

@milindsrivastava1997 milindsrivastava1997 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zzylol
The differential workflow is currently deterministically red for two independent reasons.

  1. ComposeLifecycle.Start starts the one-shot planner with docker compose up -d …, then separately calls docker compose wait planner. planner can successfully exit in the interval between those commands, leaving wait with no container to observe (no containers for project ...). This already prevents the existing corpora from running. Please start and await the planner atomically (for example, an up invocation that waits and propagates the planner exit code), or otherwise synchronize on the generated snapshot.

  2. The new issue-754 corpus contains quantile_over_time(0.9, data[1m]) / quantile_over_time(0.5, data[1m]), which the selected-plan gate explicitly classifies as requiring exact_fallback/. The subsequent benefit measurements therefore can never run, and the required CI job stays red. This is a valid semantic gate result, but it means the PR is not merge-ready as a level-3 benchmark: implement local execution for that query, or remove/mark it as unsupported until that work lands.

@zzylol
zzylol force-pushed the test/issue754-level3 branch from 4c91b8d to c338080 Compare September 24, 2026 12:55
@zzylol
zzylol changed the base branch from issue-752 to 732-test-add-prometheus-remote-write-promql-differential-suite September 24, 2026 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants