Skip to content

feat: centralize estimator accuracy and source-driven HLL sizing - #460

Merged
zzylol merged 6 commits into
mainfrom
feat/issue-459-hll-confidence
Sep 24, 2026
Merged

zzylol merged 6 commits into
mainfrom
feat/issue-459-hll-confidence

Conversation

@zzylol

@zzylol zzylol commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Before this PR

Generic HLL exposes an RSE magnitude with unknown failure probability. Even a source with an enforced bounded distinct population cannot obtain a certified HLL candidate through the default Planner workflow. Deployments would have to wire estimator confidence and precision sizing themselves.

After this PR

Accuracy code is organized under asap-aware-mapping::accuracy: public interfaces, evidence, composition, allocation, reconciliation, and estimators with separate KLL, DDSketch, HLL, CMS, CountSketch, KMV/Theta and UnivMon implementations. Estimator modules own their guarantee and sizing formulas. There is no separate HLL candidate-selection rule or compatibility module at the former paths.

Deployments provide AccuracyEvidenceProvider::estimator_contract for the complete aggregate expression. Planner combines this scoped evidence with the query or allocated local accuracy target, sizes HLL, derives its readout guarantee, and applies the existing propagation and candidate-legality checks. A deployment-specific cost or accuracy model is not needed to enable this path. The evidence includes the complete population across all merged panes; observations and ERP measured maximum errors are not proofs.

For example, a trusted classic-HLL source bound of 128 distinct values with epsilon 0.05 and delta 0.01 produces a confidence-sized HLL candidate using only default models and a source evidence provider. Missing evidence remains uncertified; invalid bounds, unsupported grouping, infeasible targets and evidence for another expression cannot authorize the result.

Accuracy architecture and source-contract responsibilities.

Estimator scope

The bounded classic-HLL model covers the linear-counting branch under independent uniform bucket hashing, maxima of 1..4096 distinct values per complete readout, and precision 4..18. It bounds collision arrivals for every integer population in the domain and deterministically bounds overestimation. It is not a Gaussian/RSE conversion or a universal HLL theorem. HIP/MLE, adversarial hashes, unbounded populations and simultaneous dashboard-wide confidence are outside this contract.

The source owner must establish the estimator, hashing and population assumptions. Planner owns the mathematical interpretation and selection semantics, not enforcement inside an external data source.

Validation

  • The evidence-only HLL regression failed before Planner integration and passes after it. Tests check precision, guarantee satisfaction, missing/wrong-scope evidence, invalid populations and an infeasible confidence target.
  • Mapping: 435 unit tests, 9 physical-handoff tests and 8 storage-I/O tests passed.
  • Exact occupancy-distribution tests independently check the probability bound in small domains; native Classic HLL split/merge tests check estimator compatibility.
  • Workspace/all-target strict Clippy, formatting and whitespace checks passed.
  • Rebased onto main at 8b9049f. Adapted the new Cardinality input list and corrected main's stale ImplementError reference to RealizationError so the rebased workspace compiles.

Human review — do not complete with an agent

  • The estimator/domain boundary is appropriate.
  • The finite probability derivation and numerical conservatism are adequate.
  • Human reviewer:
  • Decision and rationale:

The design document also includes two grouped p95 consumers sharing the tighter rank-error guarantee, with examples of semantic and confidence differences that prevent this reconciliation. Existing accuracy tests moved with their responsible modules; the refactor preserves formulas and selection semantics. An additional strict rustdoc check remains blocked by pre-existing broken links/private links and HTML warnings elsewhere in the crate.

@zzylol
zzylol force-pushed the feat/issue-459-hll-confidence branch from cd7e9e0 to 7796681 Compare September 24, 2026 14:28
@zzylol zzylol changed the title feat: add bounded classic HLL confidence and precision sizing feat: centralize estimator accuracy and source-driven HLL sizing Sep 24, 2026
@zzylol
zzylol merged commit 6ecdf88 into main Sep 24, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant