feat: centralize estimator accuracy and source-driven HLL sizing - #460
Merged
Merged
Conversation
This was referenced Sep 23, 2026
zzylol
force-pushed
the
feat/issue-459-hll-confidence
branch
from
September 24, 2026 14:28
cd7e9e0 to
7796681
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this PR
Generic HLL exposes an RSE magnitude with unknown failure probability. Even a source with an enforced bounded distinct population cannot obtain a certified HLL candidate through the default Planner workflow. Deployments would have to wire estimator confidence and precision sizing themselves.
After this PR
Accuracy code is organized under
asap-aware-mapping::accuracy: public interfaces,evidence,composition,allocation,reconciliation, andestimatorswith separate KLL, DDSketch, HLL, CMS, CountSketch, KMV/Theta and UnivMon implementations. Estimator modules own their guarantee and sizing formulas. There is no separate HLL candidate-selection rule or compatibility module at the former paths.Deployments provide
AccuracyEvidenceProvider::estimator_contractfor the complete aggregate expression. Planner combines this scoped evidence with the query or allocated local accuracy target, sizes HLL, derives its readout guarantee, and applies the existing propagation and candidate-legality checks. A deployment-specific cost or accuracy model is not needed to enable this path. The evidence includes the complete population across all merged panes; observations and ERP measured maximum errors are not proofs.For example, a trusted classic-HLL source bound of 128 distinct values with epsilon 0.05 and delta 0.01 produces a confidence-sized HLL candidate using only default models and a source evidence provider. Missing evidence remains uncertified; invalid bounds, unsupported grouping, infeasible targets and evidence for another expression cannot authorize the result.
Accuracy architecture and source-contract responsibilities.
Estimator scope
The bounded classic-HLL model covers the linear-counting branch under independent uniform bucket hashing, maxima of 1..4096 distinct values per complete readout, and precision 4..18. It bounds collision arrivals for every integer population in the domain and deterministically bounds overestimation. It is not a Gaussian/RSE conversion or a universal HLL theorem. HIP/MLE, adversarial hashes, unbounded populations and simultaneous dashboard-wide confidence are outside this contract.
The source owner must establish the estimator, hashing and population assumptions. Planner owns the mathematical interpretation and selection semantics, not enforcement inside an external data source.
Validation
8b9049f. Adapted the new Cardinality input list and corrected main's staleImplementErrorreference toRealizationErrorso the rebased workspace compiles.Human review — do not complete with an agent
The design document also includes two grouped p95 consumers sharing the tighter rank-error guarantee, with examples of semantic and confidence differences that prevent this reconciliation. Existing accuracy tests moved with their responsible modules; the refactor preserves formulas and selection semantics. An additional strict rustdoc check remains blocked by pre-existing broken links/private links and HTML warnings elsewhere in the crate.