diff --git a/docs/design_docs/asapquery-compatibility-profile.md b/docs/design_docs/asapquery-compatibility-profile.md index e911510f..608c2909 100644 --- a/docs/design_docs/asapquery-compatibility-profile.md +++ b/docs/design_docs/asapquery-compatibility-profile.md @@ -1,452 +1,277 @@ # ASAPQuery compatibility profile -> Status: implemented; wider maintenance capabilities retain separate admission gates +> Status: implemented; wider maintenance capabilities retain separate admission gates. > > Reference: [ProjectASAP/ASAPQuery at `9fb051a`](https://github.com/ProjectASAP/ASAPQuery/tree/9fb051aa798361fca8e3012835412cb6fa338a0c) -> -> Scope: a deliberately smaller target operating profile of ASAPQuery-backend that -> accepts raw Prometheus Remote Write samples and does not depend on -> ASAPCollector. -## Goal +The `asapquery` profile is a strict, startup-validated subset of +ASAPQuery-backend. It accepts raw Prometheus Remote Write samples, builds +summaries in process, serves supported PromQL and falls back to Prometheus. It +does not depend on ASAPCollector. + +The profile is implemented. The state-slot, separate runtime-inventory and +Planner handoff descriptions below are [proposed contracts](asapplanner-integration.md), +not claims that those fields already exist in the running backend. + +## Document map + +1. [Profile at a glance](#profile-at-a-glance) +2. [Worked example](#worked-example) +3. [Profile boundary](#profile-boundary) +4. [Planning and runtime contracts](#planning-and-runtime-contracts) +5. [Remote Write contract](#remote-write-contract) +6. [Query and fallback contract](#query-and-fallback-contract) +7. [Activation and readiness](#activation-and-readiness) +8. [Compatibility evidence](#compatibility-evidence) +9. [Completion and extensions](#completion-and-extensions) + +## Profile at a glance + +```mermaid +flowchart LR + P[Prometheus] -->|remote_write| I[Backend ingest] + I --> C[In-process precompute] + C --> S[SummaryStore] + U[PromQL client] --> Q[Query router] + Q -->|ready and supported| S + Q -->|fallback| P +``` -ASAPQuery-backend has a broader architecture than ASAPQuery: it can coordinate -external collectors, accept materialized summaries over modified OTLP, use -multiple storage tiers, and compile distributed physical plans. This profile -does not remove those capabilities. It defines the target configuration required -for ASAPQuery-compatible behavior and gives that configuration an independent -end-to-end acceptance target. Selecting `--profile asapquery` is now a strict, -startup-validated subset of the broader runtime. +Prometheus remains the exact raw-data authority. The backend accelerates only +installed queries whose summary state is complete, fresh and accurate enough. +Unsupported, unsafe, stale or not-yet-ready queries follow the explicit fallback. -The user-visible goal is the same drop-in shape as ASAPQuery: +Planning uses configured `QueryWorkload` and `DataWorkload`, canonical +ASAPPlanner selection and a backend-only physical compiler: ```text -Prometheus ── remote_write raw samples ──► ASAPQuery-backend - ▲ │ - │ exact fallback │ in-process summaries - │ ▼ -Grafana / PromQL client ── query ──► accelerated query endpoint +workloads -> ASAPPlanner -> selected post-ASAP computation and producer deployments + -> backend physical compiler + -> Summary Catalog definitions + PrecomputePlan + QueryPlan ``` -Prometheus remains the exact raw-data system. It sends a copy of ingested -samples to ASAPQuery-backend through Prometheus Remote Write. The backend builds -planned summaries from those raw samples and serves supported PromQL from the -summaries. Unsupported, not-yet-ready, stale, or unsafe queries are forwarded -to Prometheus. - -The reference behavior is anchored by ASAPQuery's -[top-level architecture](https://github.com/ProjectASAP/ASAPQuery/blob/9fb051aa798361fca8e3012835412cb6fa338a0c/README.md), -[Remote Write decoder](https://github.com/ProjectASAP/ASAPQuery/blob/9fb051aa798361fca8e3012835412cb6fa338a0c/asap-query-engine/src/drivers/ingest/prometheus_remote_write.rs), -[query tracker](https://github.com/ProjectASAP/ASAPQuery/blob/9fb051aa798361fca8e3012835412cb6fa338a0c/asap-query-engine/src/query_tracker/tracker.rs), -and -[precompute design](https://github.com/ProjectASAP/ASAPQuery/blob/9fb051aa798361fca8e3012835412cb6fa338a0c/asap-query-engine/src/precompute_engine/precompute_engine_design_doc.md). -The goal is behavioral compatibility, not copying its historical planner or -internal types. - -## Profile boundary +In the proposed SDS contract, the compiler binds each stored producer's writer +and query readers to the same definition and plan-scoped state slot. The plans +and catalog bindings install as one `plan_version`; publishing a new state +instance does not change that version. +The current backend obtains producer lifecycle decisions during physical +compilation. The [target integration](asapplanner-integration.md) passes the +selected computation and its associated deployment decisions together, while +retaining the complete query root and shared producer identity. + +## Worked example + +An operator configures a repeating five-minute sum query and Prometheus sends raw +samples to the backend: + +```yaml +profile: asapquery +prometheus_fallback: http://prometheus:9090 +query_workload: + - id: api-request-sum + expression: sum_over_time(api_requests_total[5m]) + every: 1m +data_workload: + metric: api_requests_total + ingestion_rate: 10000 +``` -### Required in this profile - -- Prometheus Remote Write v1 ingestion at `POST /api/v1/write`; -- Snappy decompression and protobuf `WriteRequest` decoding; -- raw scalar sample, stale-marker, and label canonicalization; -- in-process streaming precompute with windowing and lateness handling; -- an in-process summary store sufficient for the accelerated query path; -- Prometheus-compatible instant and range query endpoints; -- startup-supplied `QueryWorkload` and `DataWorkload` snapshots and canonical - ASAPPlanner invocation; -- backend-only physical compilation and atomic plan activation; -- summary-backed execution with explicit Prometheus fallback; and -- one self-contained compatibility demo and end-to-end test. - -### Excluded from the MVP profile - -- ASAPCollector discovery, configuration, or plan publication; -- `CollectorPlan`, `SDKPlan`, OpAMP, or collector activation evidence; -- OTLP or modified-OTLP ingestion; -- prebuilt summary ingestion from external producers; -- source sampling, GOS, sparse delta transmission, and frame ACK protocols; -- VictoriaMetrics Remote Write, Kafka, CSV, JSON, or other ingest connectors in - the MVP; -- SQL, ClickHouse, and Elasticsearch query protocols in the MVP; -- S3, Thanos, Gorilla, or other durable/cold tiers; and -- distributed placement or multi-producer summary merging. - -Excluded features may continue to exist in the larger product. They must be -disabled and must not be startup dependencies when this profile is selected. - -## Target architecture +Startup compiles one backend-local producer and its query readout, installs +inactive query routes and begins accepting Remote Write. The five-minute input +range and one-minute query cadence are distinct from the producer's physical +pane width, refresh schedule and state retention. The selected deployment must +provide state for each required query endpoint; the backend cannot infer that +schedule from the query DAG alone. Before complete coverage for an endpoint, the +query is forwarded to Prometheus. Once a matching state instance is ready, the +same request is served from its payload. An unsupported expression such as +`absent(up)` continues to fall back. ```text -Runtime data path - -Prometheus ── remote_write ──► raw receiver ──► precompute ──► SummaryStore - -PromQL client ──► Prometheus API ──► query router - ├──► summary readout ◄── SummaryStore - └──► exact fallback ──► Prometheus - -Planning path - -configured QueryWorkload + DataWorkload - │ - ▼ - ASAPPlanner - Post-ASAP candidates + selection - │ - ▼ - backend-only physical compiler - │ │ │ - ▼ ▼ ▼ - PrecomputePlan SummaryCatalog QueryPlan DAG - │ │ │ - ▼ ▼ ▼ - precompute catalog SID-bound executor +request -> active QueryPlan? + -> supported binding? + -> complete and fresh state? + yes: summary result + no: semantically equivalent Prometheus request ``` -The plan has no collector projection. One compile produces an atomic -`PhysicalPlan` containing one `SummaryCatalog` plus sibling backend-local -`PrecomputePlan` and `QueryPlan` sections from the same selected Post-ASAP candidate. -`PrecomputePlan` directly is the runtime precompute contract; there is no second -streaming-config semantic model or lossy conversion step. All sections share -plan and materialization identities. -One immutable version installs the precompute configuration, store catalog, and -inactive query routes atomically. Materialization readiness is runtime state: -each route becomes eligible for summary serving only after its required windows -have complete and fresh coverage. +## Profile boundary -## Component responsibilities +Required: -### ASAPPlanner +- `POST /api/v1/write` Remote Write v1 with Snappy/protobuf decoding; +- raw scalar samples, stale markers and canonical labels; +- backend-local streaming precompute, windowing and lateness handling; +- in-process SummaryStore and Prometheus-compatible instant/range endpoints; +- startup workload snapshots, canonical Planner invocation and physical compile; +- atomic activation, readiness gating and explicit Prometheus fallback; +- a self-contained compatibility demo and end-to-end test. -ASAPPlanner consumes the complete query workload and associated data workload. -It owns query semantics, workload-wide sharing, abstract summary candidates, -accuracy reasoning, logical window/lifecycle choices, selection, and exact -fallback decisions. +Disabled and not startup dependencies: -The compatibility profile consumes canonical types and behavior from the pinned -ASAPPlanner revision. It must not restore ASAPQuery's historical planner as a -second planner or copy Planner optimizer, intent-algebra, or sketch-capability -rules into ASAPQuery-backend. +- ASAPCollector, CollectorPlan, SDKPlan, OpAMP and collector activation; +- OTLP/prebuilt-summary ingestion and external summary producers; +- sampling, GOS, sparse delta transmission and frame ACK protocols; +- non-Prometheus ingest adapters and non-PromQL query protocols; +- durable/cold tiers and distributed placement or merging. -### ASAPQuery-backend control plane +These capabilities may exist in broader ASAPQuery-backend profiles. -For the first MVP, operators provide immutable `QueryWorkload` and -`DataWorkload` snapshots at startup. The control plane: +## Planning and runtime contracts -1. validates and loads both configured workload snapshots; -2. invokes ASAPPlanner with the complete `QueryWorkload` and associated - `DataWorkload`; -3. enumerates only backend-local implementations for Planner candidates; -4. returns implementation-cost evidence needed for selection; -5. compiles the selected candidate into a matching SummaryCatalog, PrecomputePlan, - and QueryPlan; and -6. stages and atomically activates those views. +| Component | Responsibility | +| --- | --- | +| ASAPPlanner | Query semantics, sharing, abstract candidates, accuracy and selected producer deployment decisions | +| Control plane | Load workloads, advertise backend-local implementations/costs, validate selected guarantee and schedule/retention, then compile and activate one physical version | +| Remote Write adapter | Decode and validate wire input; emit canonical raw samples | +| Precompute runtime | Route series, maintain windows/accumulators and publish state | +| SummaryStore and runtime inventory | Current store tracks state and readiness; proposed SDS separates payload bytes from instance metadata for partition, coverage, format and readiness | +| Query path | Execute the installed QueryPlan or forward an equivalent request to Prometheus; proposed SDS resolves ready instances through bound state references | -It does not enumerate SDK or Collector placements in this profile. A Planner -candidate that has no backend-local SummaryStore materialization and readout -implementation is unavailable; the control plane must not assign it an -optimistic zero cost. +The backend does not restore ASAPQuery's historical planner or precompute engine. +Historical fixes remain a migration source and require regression tests or an +explicit inapplicability record. The audit includes idle/trailing-window closure, +wall-clock safety, pane eviction, millisecond windows, value routing, +CMS-with-heap parameters and accumulator routing. -Online query observation, data-workload estimation from Remote Write, and -replanning may be added later. If enabled, the data-plane query endpoint emits -bounded canonical observations and the control plane aggregates them; the -request path never owns planning. These capabilities are not startup or MVP -dependencies. +A candidate without backend-local producer and readout support is unavailable; +it never receives optimistic zero cost. If Planner selects no feasible +maintenance guarantee for a required producer, the backend must use an explicit +supported fallback or reject the plan. Request handling does not perform +planning or choose a substitute producer. -### Raw ingest and precompute +## Remote Write contract -The Remote Write adapter owns only wire decoding and validation: +The adapter performs: ```text -HTTP body - -> verify request limits and content encoding - -> Snappy decode - -> protobuf WriteRequest decode - -> validate __name__, labels, timestamps, and sample encodings - -> recognize the Prometheus stale-NaN marker before ordinary numeric checks - -> canonical series key - -> route raw samples to the active PrecomputePlan +limits -> Snappy decode -> WriteRequest decode -> sample/label validation + -> stale-marker recognition -> canonical series key -> active plan routing ``` -The precompute engine owns series routing, bounded buffering, watermark and -lateness behavior, window assignment, selected accumulator updates, and writes -to SummaryStore. The decoder must not choose an aggregation family. - -The current ASAPQuery-backend streaming precompute engine is the destination, -but the ASAPQuery history is a required bug-fix migration source. Before the -compatibility profile is complete, relevant ASAPQuery precompute fixes must be -audited commit by commit and either migrated with regression tests or recorded -as inapplicable because the affected feature was intentionally retired. The -known audit set includes idle/trailing-window closure, active-ingest wall-clock -safety, move-out at pane eviction, millisecond windows, value-column routing, -CMS-with-heap parameter parsing, and accumulator-family routing. - -Restoring Remote Write must add an adapter to the current engine; it must not -restore or fork the historical ASAPQuery engine. In particular, retired -`SetAggregator` and `DeltaSetAggregator` paths are not restored merely because -their historical patches touched precompute code. Compatibility tests preserve -both migrated ASAPQuery fixes and newer backend fixes, and expose any remaining -semantic mismatch rather than weakening the expected results. - -The exact Prometheus stale-NaN bit pattern is recognized, deduplicated and -counted, then excluded from numeric aggregation. It is not currently propagated -as a worker lifecycle event. Other non-finite values, native histograms and -exemplars are rejected. Only Remote Write v1 is supported. - -Within the configured in-memory deduplication horizon, an identical canonical -series/timestamp/value is a duplicate; a conflicting value is rejected. The -receiver validates the batch and reserves all bounded worker-queue capacity -before exposing messages. A `204` acknowledges that queue admission and -in-memory dedup bookkeeping, not completed accumulator mutation or durable -commit. There is no durable raw-sample WAL or receiver dedup recovery after a -process restart. Summary persistence is a separate later stage. - -Invalid batches fail before enqueueing. Queue/dedup capacity exhaustion returns -a retryable `503`; body/decoded size limits return `413`. The dedup horizon must -cover configured lateness and expected retry duration, but it does not provide -crash-safe exactly-once delivery. Prometheus remains the raw-data authority. - -Remote Write carries neither a producer-partition roster nor an authoritative -watermark. Finite `POST /api/v1/precompute/drain` closes the input generation; -it is not continuous completion. Existing typed `SummaryWatermarkBarrier` and -coordinator APIs need registered producer/partition identity and publication -ordering before they can establish continuous closure. There is no public HTTP -barrier endpoint in this profile. - -### SummaryStore - -The MVP may use the existing in-process summary store. It stores only state -produced by the active backend-local PrecomputePlan and indexes it by -materialization, canonical label values, and logical window. It tracks enough -coverage and watermark state to distinguish ready, missing, incomplete, and -stale ranges. - -Persistent/cold storage is an extension of ASAPQuery-backend, not a dependency -of this compatibility profile. A process restart may lose the warm summary -cache only if query routing falls back to Prometheus until the new plan has -rebuilt complete coverage. Recovery creates a new backend-local producer/runtime -epoch. Pre-crash partial windows remain incomplete and must never be combined -with post-restart samples; they can be served only after explicit Prometheus -backfill, otherwise the first eligible result is a complete post-restart window. - -### Query path - -The public MVP query surface is: +Only v1 scalar samples are supported. The exact Prometheus stale-NaN marker is +recognized, deduplicated and counted, then excluded from numeric aggregation. +Other non-finite values, native histograms and exemplars are rejected. -```text -GET or POST /api/v1/query -GET or POST /api/v1/query_range -``` +Within the in-memory dedup horizon, identical series/timestamp/value input is a +duplicate and a conflicting value is rejected. The receiver validates the batch +and reserves all queue capacity before enqueueing. A `204` acknowledges admission +and dedup bookkeeping, not accumulator mutation or durable commit. -For each request, the Prometheus adapter preserves query semantics and response -shape. Semantic preservation includes `query`, `time`, `start`, `end`, `step`, -and `timeout`, plus configured tenant and authorization context; it does not -require byte-for-byte reproduction of the incoming HTTP request. Routing has two -successful outcomes: +| Condition | Result | +| --- | --- | +| Invalid batch | Fail before enqueueing | +| Queue/dedup capacity exhausted | Retryable `503` | +| Body or decoded size limit exceeded | `413` | +| Process restart | No raw WAL or dedup recovery; Prometheus remains authority | -1. execute the active QueryPlan DAG, whose materialization bindings were - resolved through SummaryCatalog, when compatible summary state has - complete and fresh coverage; or -2. forward a semantically equivalent request to the configured Prometheus - endpoint. +Remote Write provides no producer roster or authoritative watermark. Finite +`POST /api/v1/precompute/drain` closes one input generation; it does not prove +continuous completion. See [summary completeness](continuous-summary-completeness.md). -A parse failure, store miss, unsupported expression, inactive plan, incomplete -window, stale coverage, or insufficient accuracy is not an empty successful -summary result. It follows the explicit fallback route or returns an error if -fallback is unavailable. +## Query and fallback contract -The first compatibility level is intentionally explicit rather than claiming -all PromQL. It supports raw scalar samples, canonical label grouping, tumbling -window sum, Prometheus `rate` and `increase` over counters, one sketch-backed -quantile operation, and instant and range evaluation of those planned -summaries. Selectors or expressions outside that set exercise the exact -Prometheus fallback path. Expanding the accelerated surface requires a versioned -compatibility-level change and conformance tests. - -## Planning and activation lifecycle - -Startup is fallback-safe: +The public surface is: ```text -start backend - -> verify Prometheus fallback health - -> load configured QueryWorkload and DataWorkload snapshots - -> run Planner candidate search and selection - -> compile one PhysicalPlan (SummaryCatalog + PrecomputePlan + QueryPlan) - -> atomically install precompute + catalog + inactive routes under one version - -> accept Remote Write and forward every query to Prometheus - -> enter Materializing state - -> wait for complete summary coverage - -> mark each ready materialization Serving +GET or POST /api/v1/query +GET or POST /api/v1/query_range ``` -The lifecycle is `Compiled -> Installed -> Materializing -> Ready -> Serving`. -Installation is the atomic configuration boundary; readiness is evidence about -runtime data coverage. Plan replacement builds a new immutable snapshot. -Precompute, store metadata, and query routing must never observe a mixture of -old and new aggregation parameters. Until the new plan is installed and warm, -the previous compatible route or Prometheus remains authoritative. - -The first MVP plans once from the startup snapshots. Runtime observation and -repeated replanning are optional, but any later implementation must preserve the -same atomic cutover and warmup rules. - -## Relationship to the broader backend - -| Concern | ASAPQuery compatibility profile | Broader ASAPQuery-backend | -| --- | --- | --- | -| Ingest source | Prometheus Remote Write raw samples | Collector materializations over modified OTLP and other explicit profiles | -| Summary construction | Backend-local only | Collector or backend placement | -| Physical outputs | SummaryCatalog + PrecomputePlan + QueryPlan | SummaryCatalog/CollectorPlan/PrecomputePlan/TransmissionPlan/QueryPlan views as applicable | -| Query protocol | Prometheus HTTP / PromQL | Additional protocols may be supported | -| Exact fallback | Upstream Prometheus | Prometheus or another compiled storage/query route | -| Storage required for MVP | In-process warm summary state | Warm, durable, archive, and remote tiers | -| Sampling and delta | Disabled | Optional physical mechanisms | - -The compatibility profile is a restricted runtime configuration. Every enabled -component belongs to ASAPQuery-backend; broader distributed features remain -available only outside this profile. - -The profile is also not a literal subset of historical ASAPQuery internals. It -preserves the relevant external behavior while adding the current canonical -ASAPPlanner types, versioned physical compilation, readiness evidence, and -stronger activation and retry contracts. - -## Implementation status - -| Area | Implemented contract | Executable evidence | -| --- | --- | --- | -| Startup/profile | Collector-free startup, excluded-component validation, fallback health gate | `data_plane` profile tests and production-process E2E | -| Physical planning | Canonical workload snapshot to one atomic SummaryCatalog/PrecomputePlan/QueryPlan bundle | `compatibility_demo_snapshot_compiles_the_complete_query_matrix` | -| Remote Write | Strict v1 decoding, stale-marker exclusion, limits, bounded in-memory deduplication and backpressure | receiver unit tests plus process E2E replay/corrupt-batch assertions | -| Precompute/store | Raw samples use the planned family; first catch-up batches close all complete windows; sketch and exact payloads share canonical SID semantics | worker/store tests and four-family process matrix | -| Query execution | QueryPlan-only serving-time lookup, node-level materialization binding, generic DAG traversal, exact fallback | instant/range process matrix and fallback request capture | -| Atomic activation | Versioned stage/activate snapshot and materialization readiness state | physical-plan endpoint tests and `/physical-plan/status` assertions | -| Real deployment | Prometheus remote_write with no Collector | `./scripts/e2e.sh asapquery-demo` | - -## Implemented phases - -### Phase A: profile and startup contract - -Add an explicit `asapquery` profile with startup validation. It permits only -Prometheus fallback, Remote Write ingestion, PromQL HTTP serving, backend-local -precompute, the warm summary store, and configured `QueryWorkload` and -`DataWorkload` inputs. An excluded connector or required Collector endpoint is -a configuration error. - -Acceptance: the backend starts with no ASAPCollector or OTLP endpoint and all -queries initially reach Prometheus. - -### Phase B: Remote Write ingestion - -Implement the v1 receiver, strict resource limits, canonical label and stale -marker handling, bounded deduplication/retry behavior, visible backpressure, and -routing into the existing raw-sample precompute input. - -Acceptance: valid Snappy/protobuf batches produce the same canonical samples and -stale-marker recognition as a reference decoder; replayed whole or partial batches -converge without double-counting; corrupt, oversized, conflicting, and -overloaded requests cannot leave untracked mutations while returning success. - -### Phase C: backend-only planning - -Load the configured query and data workload snapshots, call the pinned -ASAPPlanner, enumerate backend-local implementations, and compile one atomic -PhysicalPlan with SummaryCatalog, PrecomputePlan, and QueryPlan. Do not create or -wait for CollectorPlan. - -Acceptance: captured Planner input, selected Post-ASAP candidate, -SummaryCatalog, PrecomputePlan, and QueryPlan are deterministic golden artifacts -with matching plan/materialization/window/family/parameter identities. - -### Phase D: atomic activation and warmup - -Install the precompute configuration, store catalog, and inactive query routes -atomically under one plan version. Track readiness separately for each -materialization and keep its queries on Prometheus until all required windows -are complete and fresh. - -Acceptance: injected failure at every installation boundary exposes either the -old complete configuration or the new complete configuration, never a mixed -one. An installed-but-materializing plan remains on fallback and cannot be -mistaken for a serving route. - -### Phase E: PromQL serving and fallback - -Serve the declared compatibility-level instant and range queries from planned -summaries. Forward every unsupported or unsafe request to Prometheus with -semantically equivalent parameters and configured request context, and preserve -Prometheus response types, labels, timestamps, warnings, and errors. - -Acceptance: accelerated results satisfy their declared error bound against -Prometheus, exact operations match Prometheus semantics, and fallback responses -are equivalent to direct Prometheus calls. - -### Phase F: compatibility demo - -`./scripts/e2e.sh asapquery-demo` runs Prometheus with `remote_write` configured -to the backend, starts the backend with fixed workload snapshots, sends their -corresponding repeating queries -through the backend, wait for planning and warmup, and capture route decisions -and resource measurements. - -Acceptance evidence includes: - -- Remote Write requests, samples, rejected requests, duplicates, and bytes; -- configured workload snapshots and the exact Planner input/output artifacts; -- active plan/materialization identities and activation time; -- summary versus fallback query counts; -- window coverage and end-to-end freshness; -- result error against direct Prometheus; and -- query latency, backend CPU, and backend memory before and after activation. - -The E2E query matrix includes all of the following through the backend and -compares each result with a direct request to the same Prometheus instance: +The adapter preserves `query`, `time`, `start`, `end`, `step`, `timeout`, tenant +and authorization semantics. It either executes the active QueryPlan or forwards +a semantically equivalent request to Prometheus. Under the proposed SDS contract, +the QueryPlan has a bound definition and state slot, resolves a matching ready +instance and reads its payload. Serving does not search the catalog for another +materialization. -| PromQL case | Instant `/api/v1/query` | Range `/api/v1/query_range` | -| --- | --- | --- | -| `rate(counter[window])` | Required | Required, including every returned step | -| `increase(counter[window])` | Required | Required, including every returned step | -| planned sum | Required | Required | -| planned sketch-backed quantile | Required | Required | -| unsupported expression | Exact fallback required | Exact fallback required | +Parse failure, unsupported expressions, inactive plans, store misses, incomplete +or stale windows and insufficient accuracy are never successful empty summary +results. They fall back or return an error when fallback is unavailable. -The counter fixture includes monotonic input, at least one counter reset, -irregular sample spacing, and samples near window boundaries. The assertions -cover values, labels, timestamps, result type, and range-step count. A test that -only proves that the endpoint returns HTTP success does not satisfy this matrix. - -## Post-MVP compatibility extensions +The first compatibility level covers raw scalar samples, canonical grouping, +tumbling-window sum, Prometheus `rate` and `increase`, one sketch-backed quantile, +and instant/range evaluation. Other expressions fall back. Expanding this set +requires a versioned compatibility level and conformance tests. -The first query extension should be a SQL endpoint because ASAPQuery exposed SQL -as an additional query surface. It must translate SQL into canonical Planner -query semantics and reuse the same catalog-backed QueryPlan readiness, summary store, and -exact-fallback rules; it must not introduce a second planner or separately -configured materializations. Its supported SQL subset and fallback target need -their own versioned compatibility contract and conformance cases. +## Activation and readiness -CSV and JSON ingestion can follow as convenience adapters. Each adapter is -limited to parsing and validation before emitting the same canonical raw-sample -records as Remote Write. It must not choose aggregation families or bypass the -active PrecomputePlan. These adapters remain optional and cannot become startup -dependencies of the Remote Write profile. - -## MVP completion criterion +```text +verify fallback -> load workloads -> select -> compile + -> atomically install catalog bindings + precompute + inactive query routes + -> accept writes and fall back queries + -> materialize complete coverage + -> mark each eligible route serving +``` -The executable completion command is: +The lifecycle is `Compiled -> Installed -> Materializing -> Ready -> Serving`. +Installation is the atomic configuration boundary; readiness describes runtime +coverage and format validation for a particular state instance. A replacement +never exposes mixed parameters from old and new plans. +Until the new version is warm, the previous compatible route or Prometheus is +authoritative. + +After restart, pre-crash partial windows remain incomplete and cannot combine +with post-restart samples. They require explicit backfill; otherwise serving +starts with the first complete post-restart window. + +## Compatibility evidence + +| Area | Required executable evidence | +| --- | --- | +| Startup | Collector-free profile validation and fallback health gate | +| Planning | Deterministic workload, selected candidate, catalog and two backend plans | +| Remote Write | Reference decoding, stale markers, limits, dedup and backpressure | +| Precompute/store | Planned families, complete windows and canonical state identity | +| Query | Instant/range summary execution plus captured fallback requests | +| Activation | Stage/activate failure tests and readiness assertions | +| Deployment | `./scripts/e2e.sh asapquery-demo` with no Collector | + +The acceptance matrix extends the original demo with the repeated workloads from +[issue #701](https://github.com/ProjectASAP/ASAPQuery-backend/issues/701) and +[issue #702](https://github.com/ProjectASAP/ASAPQuery-backend/issues/702). It is +required coverage, not a claim that the current demo already runs every row: + +| Query family | Required instant checks | Required range checks | Current evidence | +| --- | --- | --- | --- | +| Counter `rate` and `increase`, including resets | Compare each planned endpoint with Prometheus | Compare every returned step | Compatibility process test covers both endpoints | +| `sum_over_time`, `count_over_time`, `min_over_time`, `max_over_time`, `avg_over_time`, and grouped sum readout | Confirm a warm result at successive evaluation times and exact semantics | Compare every step, including non-aligned endpoints | #701/#702 process test covers successive instant evaluations; complete range matrix remains to be added | +| Instant `sum`, `count`, and `avg`, grouped by `job` and ungrouped | Compile or explicitly route to exact; compare series membership and values | Compare every step of a range request for the same expression | #702 process test covers successive instant evaluations; range comparison remains to be added | +| `quantile_over_time` with 15m range every 1m and 5m range every 30s or 10s; instant `quantile by (job)` every 1s | Confirm shared producers, warm non-aligned endpoints and the declared accuracy guarantee | Compare every step, including endpoints between physical pane boundaries | #701/#702 process test covers successive instant evaluations; full range comparison remains to be added | +| `topk` over temporal sum and count | Compare values, labels and changing Top-K membership | Compare membership and values at every step | Compatibility process test covers two range steps | +| Quantile ratio and `avg_over_time`/quantile ratio | Check composed result when defined; require exact fallback when the denominator makes the promised error undefined | Apply the same rule at every step | #701/#702 process test covers instant composition and zero-denominator fallback; range comparison remains to be added | +| Unplanned or unsupported expression | Forward the complete request to Prometheus | Forward the complete range request | Compatibility process test captures both fallback requests | + +The same input must reach backend and Prometheus for a real differential run. +Exact results compare values, labels, timestamps and result shape; approximate +quantiles use their declared error guarantee rather than byte equality. A warm +claim must be checked separately from numeric parity, and a fallback claim must +be backed by a captured upstream request. Counter fixtures cover reset, +irregular spacing and boundary samples. Range assertions include step count and +every returned evaluation, not just the first and last. + +The current #701/#702 process fixture forces a fully warm candidate with +synthetic correctness quotes. Its real Prometheus oracle runs only when +`ASAP_CURRENT_SERIES_PROMETHEUS_URL` is set; otherwise it proves warm routing and +fixture assertions without a real differential comparison. The Docker demo +covers a smaller set of queries. Neither run alone satisfies this matrix. +Record workload and Planner artifacts, active IDs, route decisions, coverage, +freshness, error, latency, CPU and memory with the completed run. + +## Completion and extensions + +The existing real-Prometheus demo command is: ```bash ./scripts/e2e.sh asapquery-demo ``` -It starts Prometheus and ASAPQuery-backend without ASAPCollector, ingests only -through Prometheus Remote Write, plans from the configured workloads, activates a -backend-local summary, serves both the declared sum and sketch-backed quantile -compatibility cases plus Prometheus `rate` and `increase` through both instant -and range endpoints from complete summary windows, and transparently falls back -for an unsupported query. The run must fail if ingestion, planning, activation, -coverage, accuracy, counter-reset handling, range-step equivalence, or fallback -evidence is missing. +It exercises the original compatibility subset. Full completion also requires +the #701/#702 workload suite with a real Prometheus oracle and the range checks +identified above. Until those checks are wired into a required gate, the demo +must not be reported as proof of the full matrix. Starting components or exposing +`/api/v1/write` alone is not completion. -Starting the components or exposing `/api/v1/write` alone is not completion. +SQL is the first intended query extension. It must translate into canonical +Planner semantics and reuse the same catalog, readiness, store and fallback +contracts. Optional CSV/JSON adapters may emit the same canonical raw-sample +records but cannot choose aggregations or bypass PrecomputePlan. diff --git a/docs/design_docs/continuous-summary-completeness.md b/docs/design_docs/continuous-summary-completeness.md index 34549b8d..c40cea0c 100644 --- a/docs/design_docs/continuous-summary-completeness.md +++ b/docs/design_docs/continuous-summary-completeness.md @@ -1,15 +1,134 @@ # Summary publication completeness for accepted input -The SummaryStore owns an in-memory admission inventory keyed by catalog generation, summary definition, group population and physical time window. Remote Write reserves every worker queue slot before admitting any coordinates, then sends the same immutable input revision with the queued work. Queue rejection leaves neither messages nor admission records behind. +Status: implemented in-memory admission and read-fence contract. Audience: +developers changing Remote Write admission, maintenance publication, SummaryStore +reads or recovery. -Workers carry the first and last consumed input revisions through materialized source and maintenance DAG outputs. State writes validate the generation and consumed revisions before modifying SummaryStore; a later output cannot acknowledge an earlier missing update. Corrections within one admitted coordinate are combined before publication. Dropped admitted input leaves the coordinate incomplete. Accepted raw-sample throughput counts once per request, without materialization fanout; materialized-output throughput counts new successful store publications, without receipt retries. +## Document map -A query rejects overlapping admitted but unpublished input. The whole QueryPlan DAG is fenced by the store revision, so branches cannot combine different publication snapshots. Both direct sketch writes (including OTLP) and exact-state writes (including SQL backfill) participate in the mutation fence; in-flight writes cannot certify a read snapshot. Finite absence proof also requires that every state mutation was admitted. The initial fence is global: unrelated updates can cause conservative exact fallback. Read-set-scoped revisions are required before claiming sustained continuous-query performance. +1. [Contract at a glance](#contract-at-a-glance) +2. [Worked example](#worked-example) +3. [Admission and publication](#admission-and-publication) +4. [Query completeness](#query-completeness) +5. [Finite drain](#finite-drain) +6. [Budgets, recovery and retirement](#budgets-recovery-and-retirement) +7. [Limits](#limits) -Finite drain closes input, waits for all workers and certifies that every accepted coordinate published. Only this closed-input proof can establish that a known series has no samples in a retained window. Missing state, unknown series, unknown time coverage and incomplete coordinates never imply an empty result. Live Remote Write provides no source watermark, so the backend does not infer global event-time completeness from the fastest series. +## Contract at a glance -Admission metadata has bounded coordinate, pending-revision and byte budgets. Completed receipts expire with configured materialization retention; pending work and the published prefixes of pending admissions remain protected. Already admitted slow-worker outputs can finish behind another worker's maintenance replay frontier. Unsolicited expired input and expired untagged replay remain rejected. +```text +Remote Write request + -> reserve all queue capacity + -> record one immutable admitted revision + -> workers publish outputs carrying consumed revisions + -> SummaryStore advances completeness only for published admitted work + -> QueryPlan reads one fenced store revision or falls back +``` -This inventory is not durable and does not establish exactly-once execution across crashes. Startup installs the authoritative catalog before persistence recovery, and version-3 persisted series metadata preserves summary definition identity and catalog provenance. Recovery requires the same catalog generation and leaves incompatible or legacy records unbound under an authoritative catalog. Reuse across changed catalog generations requires a separate explicit compatibility decision. General multi-input maintenance transforms and durable producer watermarks remain separate work. +An accepted request promises queue admission, not completed summary publication. +Missing or unpublished admitted work makes an overlapping query incomplete; it +never means an empty result. -Durable SID bindings also preserve retirement and expiry timestamps and a removal tombstone. Lifecycle changes publish through the same serialized metadata writer as the flusher before changing in-memory visibility. A stale flush snapshot cannot clear those fields. Recovery leaves removed and expired instances unregistered and preserves a still-retired instance's expiry deadline. Tombstones remain until durable state is explicitly reclaimed; this change does not claim automatic tombstone garbage collection or cross-generation reactivation. +## Worked example + +Assume request revision `101` contains samples for two coordinates: + +```text +(service=api, window=12:00..12:01) +(service=web, window=12:00..12:01) +``` + +The receiver reserves both worker slots before acknowledging the request. The +`api` worker publishes state carrying `first_revision=101` and +`last_revision=101`, while the `web` worker has not finished. + +```yaml +admission_revision: 101 +published: + api/12:00..12:01: 101 +pending: + web/12:00..12:01: 101 +``` + +A query covering both services at store fence `101` cannot combine the published +`api` state with a missing `web` state. It uses its configured exact fallback or +returns unavailability. After `web` publishes, the same fenced read may use both +states. If queue reservation initially fails, neither coordinate nor admission +record is created and the request is retryable. + +## Admission and publication + +The in-memory admission inventory is keyed by catalog generation, summary +definition, group population and physical time window. + +Here `catalog generation` names the current admission and recovery key. In the +[proposed SDS contract](summary-catalog-sds-architecture.md), the coherent +installed plan bundle uses `plan_version` and a plan-scoped state slot; runtime +instance metadata records actual partition, coverage, format and readiness. +Adapting persisted records requires an explicit mapping, not a silent rename of +the existing generation field. The catalog holds definition semantics rather +than per-instance publication status. + +- Remote Write reserves every required worker slot before admitting any work. +- All queued work carries the same immutable input revision. +- Queue rejection leaves no messages or admission records. +- Workers propagate first and last consumed revisions through maintenance DAG + outputs. +- State writes validate generation and revisions before mutating SummaryStore. +- Corrections within one admitted coordinate combine before publication. +- A later output cannot acknowledge an earlier missing update. + +Accepted raw-sample throughput counts once per request, without materialization +fanout. Materialized-output throughput counts new successful publications, +without receipt retries. + +## Query completeness + +A query rejects any range overlapping admitted but unpublished input. The whole +QueryPlan DAG uses one store revision so branches cannot combine different +publication snapshots. + +Direct sketch writes, including OTLP, and exact-state writes, including SQL +backfill, participate in the mutation fence. In-flight writes cannot certify a +read snapshot. Finite absence proof additionally requires that every mutation +was admitted. + +The initial fence is global, so unrelated updates may conservatively cause exact +fallback. Read-set-scoped revisions are required before claiming sustained +continuous-query performance. + +## Finite drain + +A finite drain closes input, waits for workers and verifies that every accepted +coordinate published. Only this closed-input proof can establish that a known +series has no samples in a retained window. + +Missing state, unknown series, unknown time coverage and incomplete coordinates +never imply an empty result. Live Remote Write has no authoritative source +watermark, so the fastest series cannot establish global event-time completeness. + +## Budgets, recovery and retirement + +Admission metadata has bounded coordinate, pending-revision and byte budgets. +Completed receipts expire with the selected producer's state retention; pending +work and its published prefixes remain protected. Admitted slow-worker outputs +may finish behind another worker's replay frontier. Unsolicited expired input +and expired untagged replay are rejected. + +Startup installs the authoritative catalog before persistence recovery. Version-3 +series metadata preserves summary-definition identity and catalog provenance. +Recovery binds only the same catalog generation; incompatible and legacy records +remain unbound unless an explicit cross-generation compatibility decision exists. + +Durable SID bindings preserve retirement/expiry timestamps and removal +tombstones. Lifecycle changes use the same serialized metadata writer as the +flusher before changing in-memory visibility. Stale flush snapshots cannot clear +those fields. Recovery does not register removed or expired instances and +preserves a retired instance's expiry deadline. + +## Limits + +The admission inventory is not durable and does not provide exactly-once +execution across crashes. Tombstones remain until explicit durable-state +reclamation; automatic garbage collection is not claimed. General multi-input +maintenance transforms and durable producer watermarks remain separate work. diff --git a/docs/design_docs/empirical-o11y-execution-plan.md b/docs/design_docs/empirical-o11y-execution-plan.md index 00e98b74..a625f1ed 100644 --- a/docs/design_docs/empirical-o11y-execution-plan.md +++ b/docs/design_docs/empirical-o11y-execution-plan.md @@ -1,67 +1,114 @@ # Offline evidence and o11y planning evaluation -This is the historical offline-evidence milestone plan. The current prototype -adds actual backend-only data-plane execution using the supplied OpenMetrics -dataset, as described in the [current evaluation guide](../user_guide/o11y-replay.md). -Benchmark production and execution tooling now belong to ASAPQuery-backend; -planner-only coverage and synthetic cached-result timing are not its acceptance criteria. +Status: historical offline-evidence milestone with a current backend execution +path. Audience: developers reproducing issue #322 and evaluating Planner/control- +plane decisions. -Audience: developers reproducing issue #322 and the planner/control-plane evaluation. +The current replay guide is [o11y replay](../user_guide/o11y-replay.md). Benchmark +production and execution tooling belong to ASAPQuery-backend; planner-only +coverage and synthetic cached-result timing do not satisfy this evaluation. -## Scope +## Document map -Use offline sketch-bench CPU, elapsed time, state/memory, disk (when measured), -and errors against offline ground truth. No runtime ground truth, posterior -feedback, or self-estimated accuracy is required. Offline accuracy observations -do not establish formal guarantees on unseen distributions. +1. [Evaluation at a glance](#evaluation-at-a-glance) +2. [Worked example](#worked-example) +3. [Evidence contract](#evidence-contract) +4. [Execution and ownership](#execution-and-ownership) +5. [Comparison rules](#comparison-rules) +6. [Acceptance](#acceptance) -## Execution and ownership +## Evaluation at a glance -1. Pin isolated planner and control-plane worktrees from fetched main branches. - Preserve existing worktrees, including unresolved cost-model changes. -2. Evidence agent: implement the versioned artifact, validation, compatibility - matching, public cost-model integration, and fallback/decision tests. -3. Benchmark agent: run actual sketch-bench algorithms on uniform and Zipf - inputs, preserve raw output, export artifacts with source and environment - provenance, and document reproducible commands. -4. Replay agent: run the existing o11y PromQL corpus through the backend parser, - its ASAPPlanner call, and the backend typed binder (not a planner-only entry), - export binding/fallback coverage and planning latency, and separately identify - supplemental sketch workloads. -5. Integration owner: connect compatible evidence to the downstream control - plane, validate dependency compatibility, review other agents' changes, run - integration checks, and report evidence-supported comparisons. - -Steps 2–4 run concurrently after agreeing the artifact contract. Integration -uses their completed interfaces and measurements. Query-pattern changes are -limited to demonstrated blockers with regression coverage; unsupported query -semantics remain explicit in the report. +```text +sketch-bench measurements ──► versioned evidence artifact + │ +OpenMetrics workload ──► backend planning/binding ──► selected plan or fallback + │ + ▼ + decision and coverage report +``` -## Acceptance +Evidence includes measured CPU, elapsed time, state/memory, disk when measured, +and error against offline ground truth. It does not claim runtime ground truth, +posterior feedback or formal guarantees on unseen distributions. + +## Worked example + +Suppose sketch-bench measures KLL on one million Zipf-distributed values: + +```yaml +evidence: + algorithm: {kind: kll, k: 200} + dataset: {distribution: zipf, exponent: 1.1, events: 1000000} + environment: {cpu: example-x86, revision: abc123} + measurements: + update_cpu_ns: {mean: 48, samples: 20} + readout_cpu_ns: {mean: 2100, samples: 20} + serialized_state_bytes: {mean: 4096, samples: 20} + rank_error: {p99: 0.008, samples: 20} + observed_at: 2026-09-01T00:00:00Z +``` + +For a compatible o11y query, the backend binds this evidence to a concrete KLL +candidate and may rank it below exact execution. If the query uses a different +distribution, stale evidence or unsupported parameters, matching fails and the +planner retains its non-empirical choice or exact fallback. The report records +the match, selected plan and evidence provenance; it does not report missing +exact-baseline cost as zero. + +Evidence for a query readout alone does not establish that its shared producer +can meet the selected maintenance guarantee and schedule/retention. The physical +compiler must validate the complete implementation combination before binding +the producer to a PrecomputePlan writer and QueryPlan readers. + +## Evidence contract + +Every artifact declares: -- Versioned schema and example; explicit units, algorithm/configuration, - dataset/distribution, environment, collection/validity times, sample count, - dispersion, and benchmark/model provenance. -- At least two actual sketch algorithms measured offline on uniform and skewed - inputs. CPU is distinct from elapsed time; serialization size is distinct - from heap memory and disk I/O. Unmeasured fields stay unavailable. -- Matching evidence affects a public planning cost/ranking/lifecycle boundary. - Tests cover changed decisions, missing/stale/mismatched evidence, invalid - values, and unchanged accuracy guarantees. -- The planner and control-plane evaluation preserve exact fallbacks and expose - unsupported shapes, measured provenance, and limits on benefit estimates. -- Reproducible measurement and replay commands, raw machine-readable results, - and a concise report distinguish actual measurements from modeled totals. +- schema version, units, algorithm and parameters; +- dataset, distribution and workload size; +- hardware/software environment and source revision; +- collection and validity times; +- sample count and dispersion; +- benchmark/model provenance; +- unavailable measurements explicitly. + +CPU and elapsed time are distinct. Serialized size, heap memory, disk I/O and +network bytes are distinct. Evidence affects a public cost, ranking or lifecycle +boundary only after compatibility and freshness validation. + +## Execution and ownership + +1. **Evidence producer:** run actual algorithms on uniform and Zipf inputs, + retain raw output and emit versioned artifacts. +2. **Evidence integration:** validate artifacts, match compatible records and + expose measured costs through the public cost-model boundary. +3. **Replay runner:** send the o11y PromQL corpus through the backend parser, + ASAPPlanner call and typed physical binder; record bindings and fallbacks. +4. **Integration owner:** pin revisions, run end-to-end checks and publish the + evidence-supported comparison. + +Artifact design, benchmarks and replay may proceed in parallel after agreeing on +the contract. Query-pattern changes require a demonstrated blocker and regression +coverage; unsupported semantics remain visible in the report. ## Comparison rules -Compare equivalent tasks, data, parameters, windows, horizons, and evaluation -cadences. Whole-plan estimates include build/update/readout, retained windows, -sharing, and raw residual work where evidence exists. Missing raw baseline or -physical evidence means an unavailable whole-plan speedup, not zero cost. -Microbenchmark algorithm comparisons are labeled separately. Disk usage and -network savings require their own measurements or explicit models. +Compare equivalent tasks, data, parameters, windows, horizons and evaluation +cadences. Whole-plan estimates include build/update/readout, retained state, +sharing and raw residual work where evidence exists. Algorithm microbenchmarks +remain separately labeled. + +Disk or network savings require measurements or an explicit model. Repeated-query +break-even requires compatible exact baseline, sketch maintenance and readout +measurements. Missing components make whole-plan speedup unavailable rather than +free. Offline estimates do not establish deployed end-to-end latency improvement. + +## Acceptance -Repeated-query break-even can be computed only when compatible exact baseline, -sketch build/maintenance, and readout measurements are present. It is an offline -estimate and does not establish deployed end-to-end latency improvement. +- At least two real sketch algorithms are measured on uniform and skewed inputs. +- Missing, stale, malformed and mismatched evidence cannot change a decision. +- Matching evidence changes at least one public planning decision in a test. +- Exact fallback and accuracy guarantees remain unchanged. +- Replay exposes supported bindings, unsupported shapes and provenance. +- Commands, raw machine-readable results and a concise report are reproducible. diff --git a/docs/design_docs/shape-aware-erp-v1.md b/docs/design_docs/shape-aware-erp-v1.md index 5d04d99a..a04e1870 100644 --- a/docs/design_docs/shape-aware-erp-v1.md +++ b/docs/design_docs/shape-aware-erp-v1.md @@ -1,34 +1,94 @@ # Shape-aware ERP v1 -The backend obtains an observed shape from the live runtime-samples feedback -path and asks ASAPPlanner for the nearest compatible benchmark profile. A -runtime record carries `erp_observed_shape.observation` using Planner's shared -`ErpShapeObservation`: cardinality, observed event count, candidate fits and an -optional dataset fingerprint. The wrapper also reports burst ratio. The planning request -selects the ring via `observed_shape_source`. - -The observer retains both uniform and fitted Zipf hypotheses, with total-variation -distance against the observed rank masses and a sample-count-adjusted fit score. -The score is a heuristic fit quality, not a statistical confidence interval or -sketch-error guarantee. Unnamed and mixed distributions are not forced into one -family. Planner jointly matches admissible hypotheses against ERP records. -Custom data first matches its fingerprint when available, otherwise it can use -the same bounded shape matching as synthetic data. +Status: implemented planning contract. Audience: developers producing runtime +shape observations or matching Error–Resource Profile (ERP) evidence. + +## Document map + +1. [Design at a glance](#design-at-a-glance) +2. [Worked example](#worked-example) +3. [Observation and matching contract](#observation-and-matching-contract) +4. [Cost composition](#cost-composition) +5. [Miss and fallback behavior](#miss-and-fallback-behavior) +6. [Payload migration](#payload-migration) +7. [Population isolation](#population-isolation) + +## Design at a glance + +```text +runtime samples ──► bounded shape observer ──► ErpShapeObservation + │ +benchmark ERP records ─────────────────────────────┤ + ▼ + compatible evidence or miss + │ + ▼ + Planner cost and parameter choice +``` + +The observer retains uniform and fitted Zipf hypotheses. Planner matches +admissible hypotheses to compatible benchmark records. A fit score is a heuristic +measure of shape similarity, not a confidence interval or sketch-error guarantee. + +## Worked example + +```yaml +observation: + cardinality: 10000 + observed_events: 1000000 + empirical_fingerprint: null + fits: + - family: zipf + parameters: {exponent: 1.1} + goodness_of_fit: 0.03 + confidence: 0.92 + +match_policy: + max_goodness_of_fit: 0.05 + min_confidence: 0.90 + max_cardinality_ratio: 2.0 +``` + +If a fresh KLL benchmark record covers a compatible cardinality, event count, +parameters and Zipf fit, Planner uses its measured atomic update, merge and query +costs. A poorer fit, ambiguous hypothesis or excessive cardinality distance is a +miss. Custom data first matches an exact dataset fingerprint when one is present. + +For a five-minute query using one-minute panes, five query evaluations and one +shared materialization, the abstract operation counts are: + +```text +updates = observed updates +merges = 5 * (ceil(5m / 1m) - 1) = 20 +queries = 5 +state = retained panes * 1 +``` + +Planner multiplies those counts by compatible measured atomic costs; the runtime +observation does not directly claim total latency. + +## Observation and matching contract + +`erp_observed_shape.observation` uses Planner's shared `ErpShapeObservation` and +contains cardinality, observed event count, candidate fits and an optional dataset +fingerprint. The wrapper also records burst ratio; `observed_shape_source` +selects the observation ring. + +Each fit reports family, parameters, goodness of fit and confidence. Total- +variation distance is computed against observed rank masses with sample-count- +adjusted scoring. Unnamed or mixed distributions are not forced into one family. +Equal frequencies produce only the canonical uniform fit; Zipf exponent zero must +not create false ambiguity. Key count, key length and occupied interval count are bounded. Sparse interval -IDs do not allocate a dense vector. Any overflow or cap violation permanently -invalidates that observation window; callers must start a fresh observer rather -than publishing a biased partial snapshot. Profiles with too few benchmark -events, poor fit, ambiguous confidence, or excessive cardinality/parameter -distance are misses. +IDs do not allocate dense vectors. Overflow or a cap violation invalidates the +entire observation window; callers start a fresh observer instead of publishing +a biased partial snapshot. -On a hit, empirical parameters and measured atomic costs are used. On a miss, -malformed evidence, or drift, Hybrid mode retains the theoretical parameters; -if the runtime cannot deploy them or they exceed its memory limit, compilation -chooses exact execution. Empirical-only mode fails closed. +## Cost composition -For a pane width `p`, query window `w`, query executions `q`, updates `u`, -retained panes `r`, and shared materializations `m`, Planner composes: +For pane width `p`, query window `w`, query executions `q`, updates `u`, retained +panes `r` and shared materializations `m`: ```text updates = u * m @@ -38,40 +98,46 @@ state = r * m CPU = updates*Cupdate + merges*Cmerge + queries*Cquery ``` -This separates machine-specific atomic measurement from workload-specific -window planning and makes tumbling, sliding/pane, retention, and sharing costs -auditable. - -### Observation payload migration - -Runtime producers must replace the old single `shape` object with an -`observation` containing `cardinality`, `observed_events`, `fits`, and optional -`empirical_fingerprint`. Each fit supplies `family`, `parameters`, -`goodness_of_fit`, and `confidence`; shape-match policy must also supply the fit -quality and confidence thresholds. Old payloads are rejected rather than given -invented confidence. Publish a new observation after upgrading the producer. - -The bounded observer models ranked key frequencies. Its output does not describe -the numeric spacing of KLL sample values and must not be advertised as a general -numeric-distribution observation. Equal frequencies produce the canonical -uniform fit only: Zipf exponent zero describes the same distribution and must -not create a false ambiguity. Near-uniform, genuinely distinct fits still pass -through the normal ambiguity policy. - -### Population isolation in backend-local execution - -A temporal scalar summary has one state per source series when its selected -`SummaryAgg` uses `Reduction::PerEntity`. An explicit reduction with no grouping -keys has one pooled population. These are different materializations even when -source, sketch parameters, and the visible grouping-key list are identical. - -The compiler records shared `PopulationPartitioning` metadata in the runtime -configuration and DataDescriptor. Both identities include the partitioning; -installation checks it against the bound Planner DAG. Raw ingestion uses the -full source labels for per-entity routing and the configured grouping for pooled -routing. Memory estimates count per-entity states against source cardinality. -Legacy configurations without this metadata retain their existing routing rules. - -This is a source-isolation contract, not permission to skip a maintenance update -expression. Only already-supported scalar update expressions pass the compiler's -per-entity admission check; other subDAG updates still require a real evaluator. +This separates machine-specific atomic measurements from workload-specific +window, retention and sharing decisions. +The count `m` refers to stored producer outputs, not a standalone catalog +`Materialization` object. A selected producer deployment also specifies its +maintenance mode and schedule/retention; those choices must be costed and +validated separately from the logical query range. + +## Miss and fallback behavior + +Profiles with too few events, poor fit, ambiguous confidence, stale evidence or +excessive cardinality/parameter distance are misses. Malformed evidence also +fails matching. + +- **Hybrid mode:** retain theoretical parameters; choose exact execution if the + runtime cannot deploy them or they exceed its memory limit. +- **Empirical-only mode:** fail closed when compatible evidence is unavailable. + +## Payload migration + +Runtime producers replace the old single `shape` object with `observation`, +including `cardinality`, `observed_events`, `fits` and optional +`empirical_fingerprint`. Match policy supplies fit-quality and confidence +thresholds. Old payloads are rejected rather than assigned invented confidence. + +This ranked-frequency observation does not describe numeric spacing of KLL values +and must not be advertised as a general numeric-distribution model. + +## Population isolation + +`Reduction::PerEntity` creates one temporal scalar state per source series. An +explicit reduction with no grouping keys creates one pooled population. They are +different stored producer bindings even when source, parameters and visible +grouping keys look identical. + +The compiler stores `PopulationPartitioning` in runtime configuration and +`DataDescriptor`; both identities include it. Installation checks it against the +bound Planner DAG. Ingestion uses full source labels for per-entity routing and +configured grouping for pooled routing. Memory estimates charge per-entity state +against source cardinality. + +This isolation contract does not implement arbitrary maintenance expressions. +Only supported scalar update expressions pass per-entity admission; other subDAG +updates still require an evaluator.