Skip to content

refactor!: compose grouped TopK and separate operator execution phase - #461

Merged
zzylol merged 9 commits into
mainfrom
refactor/membership-subgraph
Sep 24, 2026
Merged

zzylol merged 9 commits into
mainfrom
refactor/membership-subgraph

Conversation

@zzylol

@zzylol zzylol commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

#460 is merged. Dependency order: main → #461 → #462.

Before this PR

CandidateTopK fused candidate membership and authoritative-value ranking. Operator payloads partially encoded execution phase. Counter-weighted TopK could summarize raw counter increments instead of the query's complete windowed rate results.

After this PR

Physical computation and execution phase are separate. Candidate pruning uses a general semi-join when that optimization is needed; grouped ranking uses Sort → Limit. The dedicated CandidateTopK operator and its special edge roles are removed, with no MembershipFilter replacement or compatibility alias.

For rate/increase-weighted grouped TopK, the summary consumes finalized per-series values directly. Its keyed sum accumulation implements the inner grouped sum. This realization does not construct a separate exact grouped-score branch or require a pruning semi-join. The obsolete ResetAwareCounterDelta expression and its dedicated reranking helper are deleted.

Worked example: one CMS and candidate heap per job

topk by(job)(2, sum by(service, job)(rate(m[1m])))
  1. Compute each series' complete reset-aware rate(m[1m]) at the requested evaluation time, including its window semantics. Preserve service and job identities.
  2. Route each rate result to that job's independent CMS and candidate heap.
  3. Insert key = service, weight = rate. Different series for the same service/job accumulate into the same key. This is how sum by(service, job) maps to the summary's sum accumulation; it does not require first computing all exact grouped sums.
  4. Read candidate services and estimated accumulated scores. Internal candidate capacity K is distinct from the query's output count k = 2. Baseline sizing uses max(k, ceil(1 / epsilon)); that sizing choice is not a membership certificate.
  5. Project the typed readout back to the logical service/job/score columns, sort by the score within each job, and apply a per-job Limit of two.
job partition service key per-series rate contributions true accumulated weight
api auth 7, 5 12
api checkout 6, 3 9
api search 4 4
batch ingest 5, 3 8
batch export 6 6
batch cleanup 2 2

The last column explains the quantity being estimated; it is not a required exact aggregation stage. If the estimates preserve this ordering and the necessary identities are retained, the output services are auth and checkout for api, and ingest and export for batch. Returned scores are summary estimates, not necessarily the exact numbers above.

Per-series rate(m[1m])
  → Partition by job
  → Per-job CMS + candidate heap: item = service, weight = rate, accumulation = sum
  → Typed candidate/estimated-score readout
  → Project logical output columns
  → Sort within job → Limit 2 within job

Partition columns are resolved against the actual rate-input schema. State and readout preserve job identity. Ranking uses the aggregate result column, not the first numeric column; unresolved or ambiguous identity mappings are rejected.

Evidence-dependent candidates: preserve #455

Missing evidence does not erase a constructible weighted TopK candidate. The candidate exports symbolic requirements for the distinct partition/item population bound and TopK membership intervals/failure probability. Partial evidence resolves only the corresponding terms. Known-invalid evidence and known contributions that already violate the accuracy target remain distinct from missing evidence.

When supplied, the distinct-item bound supports simultaneous score guarantees for adaptively selected candidates and confidence sizing. The score guarantee is retained in provenance; score and membership probabilities compose through a symbolic union bound. Membership evidence refers to the query's output count, not the internal candidate capacity. Optimistic floors are used only for the existing logical target checks and are never exported as certificates.

Planner represents candidates and evidence requirements. Evidence acquisition, deployment selection and execution policy belong to the integrating backend/runtime/deployment; this PR adds no deployment admission mechanism. See the evidence-dependent candidate contract.

Execution phase

Deployment assigns ingestion time or query time independently of operator identity. The same computation supports either placement through ExecutableDag::with_execution_phases. Summary construction can consume finalized values in either phase. A semantic constructor's initial layout is not a placement restriction.

Each evaluation must represent its own complete rate window. An ingestion deployment may compute and store that window's summary; a query deployment may construct it on demand. Continuously adding successive moving-window rate snapshots to one cumulative state is not equivalent to this query.

Executable DAG wire version 4 removes the old candidate operator, special edge roles, duplicate phase fields and old phase names. Shared Float64 weighted-CMS execution and the summary_operators module live in downstream #462.

Validation

  • 438 mapping unit tests and 211 types unit tests passed.
  • PromQL integration: 28 tests passed. SQL and exact-composition suites: 12 and 14 tests passed.
  • Regressions cover finalized rate inputs, per-job partitions, independent candidate capacity/output count, grouped Sort/Limit, both phase assignments, missing/partial/invalid evidence, and symbolic evidence export.
  • The missing-population-evidence regression failed against the rejecting implementation before the fix.
  • Strict workspace/all-target/all-feature Clippy, formatting and whitespace checks passed on the updated stack.

@zzylol

zzylol commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

What is MembershipFilter?

@zzylol

zzylol commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

what do you mean by candidate pruning, and what is the nodes you modified?

@zzylol
zzylol force-pushed the feat/issue-459-hll-confidence branch from cd7e9e0 to 7796681 Compare September 24, 2026 14:28
@zzylol
zzylol force-pushed the refactor/membership-subgraph branch 3 times, most recently from 0afd38a to 36e7da7 Compare September 24, 2026 15:00
@zzylol
zzylol force-pushed the refactor/membership-subgraph branch from 36e7da7 to da6ecbf Compare September 24, 2026 15:08
@zzylol
zzylol changed the base branch from feat/issue-459-hll-confidence to main September 24, 2026 15:09
@zzylol zzylol changed the title refactor!: compose candidate pruning and separate execution phase from operators refactor!: compose grouped TopK and separate operator execution phase Sep 24, 2026
@zzylol
zzylol merged commit 8acb472 into main Sep 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant