refactor!: compose grouped TopK and separate operator execution phase - #461
Merged
Merged
Conversation
Contributor
Author
|
What is MembershipFilter? |
Contributor
Author
|
what do you mean by candidate pruning, and what is the nodes you modified? |
zzylol
force-pushed
the
feat/issue-459-hll-confidence
branch
from
September 24, 2026 14:28
cd7e9e0 to
7796681
Compare
zzylol
force-pushed
the
refactor/membership-subgraph
branch
3 times, most recently
from
September 24, 2026 15:00
0afd38a to
36e7da7
Compare
zzylol
force-pushed
the
refactor/membership-subgraph
branch
from
September 24, 2026 15:08
36e7da7 to
da6ecbf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#460 is merged. Dependency order: main → #461 → #462.
Before this PR
CandidateTopK fused candidate membership and authoritative-value ranking. Operator payloads partially encoded execution phase. Counter-weighted TopK could summarize raw counter increments instead of the query's complete windowed rate results.
After this PR
Physical computation and execution phase are separate. Candidate pruning uses a general semi-join when that optimization is needed; grouped ranking uses Sort → Limit. The dedicated CandidateTopK operator and its special edge roles are removed, with no MembershipFilter replacement or compatibility alias.
For rate/increase-weighted grouped TopK, the summary consumes finalized per-series values directly. Its keyed sum accumulation implements the inner grouped sum. This realization does not construct a separate exact grouped-score branch or require a pruning semi-join. The obsolete ResetAwareCounterDelta expression and its dedicated reranking helper are deleted.
Worked example: one CMS and candidate heap per job
rate(m[1m])at the requested evaluation time, including its window semantics. Preserve service and job identities.key = service,weight = rate. Different series for the same service/job accumulate into the same key. This is howsum by(service, job)maps to the summary's sum accumulation; it does not require first computing all exact grouped sums.Kis distinct from the query's output countk = 2. Baseline sizing usesmax(k, ceil(1 / epsilon)); that sizing choice is not a membership certificate.The last column explains the quantity being estimated; it is not a required exact aggregation stage. If the estimates preserve this ordering and the necessary identities are retained, the output services are auth and checkout for api, and ingest and export for batch. Returned scores are summary estimates, not necessarily the exact numbers above.
Partition columns are resolved against the actual rate-input schema. State and readout preserve job identity. Ranking uses the aggregate result column, not the first numeric column; unresolved or ambiguous identity mappings are rejected.
Evidence-dependent candidates: preserve #455
Missing evidence does not erase a constructible weighted TopK candidate. The candidate exports symbolic requirements for the distinct partition/item population bound and TopK membership intervals/failure probability. Partial evidence resolves only the corresponding terms. Known-invalid evidence and known contributions that already violate the accuracy target remain distinct from missing evidence.
When supplied, the distinct-item bound supports simultaneous score guarantees for adaptively selected candidates and confidence sizing. The score guarantee is retained in provenance; score and membership probabilities compose through a symbolic union bound. Membership evidence refers to the query's output count, not the internal candidate capacity. Optimistic floors are used only for the existing logical target checks and are never exported as certificates.
Planner represents candidates and evidence requirements. Evidence acquisition, deployment selection and execution policy belong to the integrating backend/runtime/deployment; this PR adds no deployment admission mechanism. See the evidence-dependent candidate contract.
Execution phase
Deployment assigns ingestion time or query time independently of operator identity. The same computation supports either placement through
ExecutableDag::with_execution_phases. Summary construction can consume finalized values in either phase. A semantic constructor's initial layout is not a placement restriction.Each evaluation must represent its own complete rate window. An ingestion deployment may compute and store that window's summary; a query deployment may construct it on demand. Continuously adding successive moving-window rate snapshots to one cumulative state is not equivalent to this query.
Executable DAG wire version 4 removes the old candidate operator, special edge roles, duplicate phase fields and old phase names. Shared Float64 weighted-CMS execution and the
summary_operatorsmodule live in downstream #462.Validation