Conversation
This reverts commit daa5281.
| ## Purpose and scope | ||
|
|
||
| ## Design decision | ||
| This design splits one selected ASAPPlanner semantic DAG into two executable |
There was a problem hiding this comment.
What is a "semantic DAG"? Is this the output of ASAPPLanner? How is this different from SummaryMaintenanceLifecyclePlan?
There was a problem hiding this comment.
“semantic DAG” means the selected post-ASAP computation DAG produced by ASAPPlanner.
SummaryMaintenanceLifecyclePlan means after a post-ASAP DAG being generated, some logic of summary maintainance will output the plan for how a summary is maintiend, incremental vs built from data at rest. So these are two steps currently in the code.
There was a problem hiding this comment.
I didn't understand the relationship between Semantic DAG and SummaryMaintenanceLifecyclePlan. Also, what is ASAPQuery-backend inputting from ASAPPlanner? One of these or both?
There was a problem hiding this comment.
this relationship will be answered in ProjectASAP/ASAPPlanner#445
There was a problem hiding this comment.
The backend uses both, at different stages:
Selected post-ASAP DAG: The backend’s selection adapter obtains a Rc root and passes it to physical compilation as QueryCompilationInput.selected_plan_root. This describes the
selected computation. CodeMaintenance lifecycle decisions: During physical compilation, select_lifecycle() calls Planner’s plan_summary_maintenance_lifecycles() with the selected summary node, workload demand,
capabilities, and cost evidence. That returns a SummaryMaintenanceLifecyclePlan. The backend extracts its selected lifecycle guarantee, window framework, implementation identity, and
costing information into backend configuration. CodeSo the current flow is:
Select post-ASAP computation
→ pass selected root to backend compiler
→ compiler calls Planner for summary lifecycle decisions
→ combine computation and selected maintenance decisions
→ generate backend plansThe lifecycle plan refers to the summary computation it is planning maintenance for. It does not replace the DAG, and the backend does not simply execute the lifecycle-plan object directly.
There was a problem hiding this comment.
Following up on my explanation above: that describes the current backend call sequence. Based on ASAPPlanner #445, I propose moving lifecycle-aware selection before physical compilation and passing its result directly to the compiler.
Current implementation
Select complete Post-ASAP DAG
-> backend physical compiler receives selected_plan_root
-> extracts summary producers
-> calls Planner lifecycle API for selected producers
-> extracts lifecycle/window decisions
-> combines those decisions with the complete query DAG
-> generates backend physical plans
The current compiler calls select_lifecycle(..., &selected.node, ...) on extracted producers (code). Those producer-local lifecycle results do not necessarily contain downstream query readouts, so the compiler still needs the separate complete query root.
Proposed integration
PlanningWorkload + evidence + capabilities
-> ASAPPlanner
-> PlanSpace
-> lifecycle-aware selection and materialization
-> SummaryMaintenanceLifecyclePlan
- complete selected Post-ASAP root
- maintenance decisions for its summary producers
- summary-versus-raw recomputation decision
-> backend physical compiler validates and binds the selected result
-> PrecomputePlan
-> QueryPlan
-> Summary Catalog definitions
-> installation and execution
| Boundary | Current implementation | Proposed integration |
|---|---|---|
| Computation supplied to compiler | Complete selected DAG root | Complete root retained in the lifecycle-aware result |
| Lifecycle selection | Called from inside physical compilation for extracted producers | Completed through Planner's helper before physical compilation |
| Connecting DAG and maintenance decisions | Backend combines the separate call results | Compiler consumes their association in the selected result |
| Compiler responsibility | Obtains lifecycle decisions and generates physical plans | Validates feasibility and generates physical plans from the selected decisions |
| Raw recomputation | Must be handled by the existing selection/compilation paths | Explicitly honor the helper's raw-versus-summary decision |
PlanSpace remains Planner's canonical logical output; lifecycle-aware selection is a helper over that output, as described in #445. The compiler still needs physical implementation, schema, storage and installation context. The simplification concerns the computation/lifecycle handoff, not removal of backend responsibilities.
Relative to the current PR #737 design, this is a smaller change: the document already requires the selected DAG plus lifecycle commitments. The proposal makes their handoff one associated result instead of independently supplied artifacts. The PrecomputePlan/QueryPlan split, catalog definitions and state references remain applicable.
The essential condition is that the result retains the complete query root, including readouts and remaining query operations. For multiple queries, preserve query-to-root associations and shared producer identity across the results so compilation does not duplicate maintenance. If raw recomputation is selected, do not unconditionally install summary producers.
This is a target integration proposal, not an interface the current backend already implements. It requires moving the lifecycle-selection call boundary and preserving those associations, not merely changing a parameter type.
|
|
||
| ## Architecture at a glance | ||
|
|
||
| The current `PrecomputePlan.executable_dags` can contain a complete semantic DAG, |
There was a problem hiding this comment.
Pls add a note that this is confusing and must be changed. Related #740
| lifecycle_commitment: | ||
| mode: batch_rebuild_from_data_at_rest | ||
| rebuild_every: 1m | ||
| retain_for: 10m | ||
|
|
||
| backend_capabilities_and_evidence: | ||
| supported_modes: [batch_rebuild_from_data_at_rest] | ||
| supported_algorithms: [kll] | ||
| kll_200_state_bytes: 4096 | ||
| five_minute_rebuild_cpu_ms: 35 |
There was a problem hiding this comment.
I do not understand these. Is there documentation?
| - id: def-api-latency-kll | ||
| input: request_latency_seconds | ||
| group_by: [service] | ||
| range: 5m |
There was a problem hiding this comment.
Examples are helpful thank you. What does range mean?
There was a problem hiding this comment.
range: 5m means the logical input window summarized by the KLL. For an evaluation at time T, it includes samples with timestamps in (T - 5m, T], grouped by service.
There was a problem hiding this comment.
So basically it's equal to the size of the time window (tumbling or sliding) used to generate the KLL instances?
| The baseline is merged code, not the completion of open PRs. | ||
|
|
||
| | Area | Existing foundation | Consolidation needed | | ||
| “Maintenance” is the execution phase that constructs or updates state, including |
There was a problem hiding this comment.
These concepts are also present in ASAPPlanner right? Wondering if these have been described there
| A legal target alternative is: | ||
| | Output | Responsibility | | ||
| | --- | --- | | ||
| | Catalog/SDS entries | Summary semantics, materialization identity, schema and state references | |
There was a problem hiding this comment.
How is this catalog actually used?
There was a problem hiding this comment.
Catalog stores the semantics / description of SDS (changed less often), the summary store/sketch store stores the SDS instances payloads (changed per instance).
There was a problem hiding this comment.
Catalog is metadata store
There was a problem hiding this comment.
I understand what it is. I am curious, how it is used right now. Is it used right now?
There was a problem hiding this comment.
catalog is being used rn.
How it's being used --
- Compilation: The physical compiler constructs SummaryCatalog from the selected precompute configurations, binds PrecomputePlan to it, and validates QueryPlan against it. Compiler code
- Runtime installation: When the startup physical plan contains a catalog, the data plane installs it into SketchStore. Startup code
- Query execution: One concrete consumer is MetricsQL per-series readout: it looks up the bound definition in the catalog, resolves its data descriptor, and restores the metric’s name
label. If that metadata is unavailable, that path requests fallback. Readout codeThe current catalog contains summary_descriptors, data_descriptors, and a materializations map. The simplified catalog described in this PR is a proposed change to that existing
representation—not the introduction of a previously unused catalog. Current type
| | Object | Meaning | Changes when | | ||
| | --- | --- | --- | | ||
| | `SummaryDefinition` | Canonical input, operation, grouping, time semantics, algorithm and parameters | Summary semantics change | | ||
| | `Materialization` | An installed decision to produce a definition with one state contract | Plan generation or physical contract changes | |
There was a problem hiding this comment.
Confused by this. I dont understand the meaning. Also, how is this "Materialization" related to the discussion at #736 (comment)
There was a problem hiding this comment.
- SummaryDefinition: summarize request_latency_seconds by service over five minutes using KLL with k=200.
- SummaryStateInstance: one concrete stored result from that producer, such as the summary for service=api covering (12:00, 12:05].
There was a problem hiding this comment.
Agreed—the standalone catalog Materialization object was an over-abstraction. I have removed it from the proposed design and examples rather than introducing another name for the same layer.
Its information now belongs to the objects that use it:
| Information | Owner |
|---|---|
| Summary meaning, state family, algorithm parameters | SummaryDefinition in the catalog |
| Plan version | The installed plan bundle; persisted instances also record it for recovery validation |
| Connection between writer and readers | A compiler-assigned state_slot_id in their StateReference, scoped to the plan version |
| Schema/encoding and physical partition rules | PrecomputePlan writer configuration and matching QueryPlan reader configuration |
| Authorized writer and maintenance policy | The PrecomputePlan binding and selected producer lifecycle |
| Physical-to-semantic provenance | The compiler's provenance mapping |
| Actual partition, coverage, readiness, location and payload format | SummaryStateInstance runtime metadata; payload bytes remain in the summary store |
The resulting flow is simply:
PrecomputePlan: Build KLL -> Write state slot S
QueryPlan: Read state slot S -> Estimate p99
Catalog: SummaryDefinition referenced by both bindings
The state slot is only a join key within a plan version, not a new catalog object with an independent lifecycle. The compiler emits both bindings from one decision and validates their agreement before installation.
Regarding #736: BackendNodeBinding::Materialization remains the existing node-placement marker meaning “store this node's output.” It does not require a separate catalog Materialization object. I have made that distinction explicit.
The docs now remove the proposed materializations collection, update the diagrams and examples, and describe migration through versioned adapters: map existing stored-output identities to slots, retain payload locators, and preserve the format/partition constraints in reader/writer bindings. Existing persisted IDs and wire fields must not be silently renamed or reinterpreted. This PR remains a design-document change; runtime migration is follow-up implementation work.
| | `SummaryDefinition` | Canonical input, operation, grouping, time semantics, algorithm and parameters | Summary semantics change | | ||
| | `Materialization` | An installed decision to produce a definition with one state contract | Plan generation or physical contract changes | | ||
| | `SummaryStateInstance` | One stored partition, such as a series/pane or completed aggregate | Runtime creates or replaces payload state | | ||
| | `StateReference` | A typed plan reference to permitted materialized state | A compiled reader/writer binding changes | |
There was a problem hiding this comment.
I did not understand the purpose of this.
There was a problem hiding this comment.
statereference is how plan can reference the summarystateinstance.
| | Selected post-ASAP DAG | Planner-selected computation graph, including summary producers, shared dependencies and query readouts. Called the “semantic DAG” in earlier discussion. | | ||
| | Summary producer | An operation or subgraph that builds summary state. Multiple queries may share its stored output. | | ||
| | `SummaryMaintenanceLifecyclePlan` | Planner result associating a post-ASAP root with deployment decisions for its unique reachable summary producers, plus workload and costing context. | | ||
| | Lifecycle commitment | Selected maintenance promise for one producer, with its scheduling and retention binding. A deployment's `SummaryMaintenanceLifecycleGuarantee` carries the Planner-level commitment. | |
There was a problem hiding this comment.
Still confused on this. Why do we need this concept?
There was a problem hiding this comment.
Good question. I used “lifecycle commitment” only as shorthand for the selected, executable maintenance decision for one summary producer: the selected SummaryMaintenanceLifecycleGuarantee together with its concrete schedule and retention binding. This distinguishes the per-producer decision from the SummaryMaintenanceLifecyclePlan, which contains the DAG, all producer deployments, alternatives, and workload/cost context. It is not a new model or API type. If the shorthand obscures that distinction, I can use “selected deployment guarantee and schedule/retention” throughout instead.
There was a problem hiding this comment.
Updated the glossary, integration design, migration plan, and conceptual YAML example to use “selected deployment guarantee and schedule/retention” instead. This is now commit ac855b2.
| window framework. The plan also carries workload demand and costing context. | ||
| Thus the lifecycle plan already refers to the computation DAG; it is not a | ||
| separate query representation, nor is one whole lifecycle plan required per | ||
| producer. A missing guarantee is not an executable maintenance commitment. |
There was a problem hiding this comment.
Did not understand last sentence on missing guarantee
There was a problem hiding this comment.
Agreed; that sentence was too compressed. I rewrote it in ea3226c. In the Planner type, a deployment’s guarantee is optional (None when no alternative is selectable). For that producer, the backend has no selected maintenance mode/schedule to implement, so it must not invent one. If a query needs that stored state, it must use an explicit supported fallback or the plan must be rejected.
| physical plans plus catalog bindings. A shared producer is maintained once for | ||
| all compatible consumers. | ||
|
|
||
| ### Lifecycle commitment |
There was a problem hiding this comment.
Very confused by this. Can you give an example situation where if I do not have this concept, there is some issue with correctness or performance or something else?
There was a problem hiding this comment.
I saw the compiler input example, but still do not understand this concept.
There was a problem hiding this comment.
You are right that the old “lifecycle commitment” wording made this sound like a separate concept. I removed that term and added a concrete example in 118e363. Suppose two queries share a five-minute KLL summary and ask for a result every minute. The selected deployment says to rebuild at each minute boundary and retain completed snapshots for ten minutes. The DAG identifies the shared computation, but by itself does not specify those maintenance decisions. If the backend independently rebuilds only every five minutes, four of five requested endpoints have no matching state; reusing an older snapshot as if it covered the requested interval would change the answer. So the need is to carry and validate the existing Planner-selected guarantee and schedule/retention, not introduce a new model or type.
| There is no separate catalog `Materialization` object. | ||
|
|
||
| The control plane reconciles two explicitly separate views: | ||
| ```mermaid |
There was a problem hiding this comment.
For the edges to SummaryDefinition catalog, which are reads and which are writes?
Related, are you saying that if I think of an SDS, the SummaryStore only stores SDS payload, while other metadata stays in the SummaryDefinition metadata catalog?
There was a problem hiding this comment.
Good catch; the arrows were ambiguous. I labeled them in 1effeb5. The compiler/catalog authority writes the SummaryDefinition when installing a plan. The PrecomputePlan and QueryPlan bindings reference that definition; those edges mean catalog reads for validation at installation, not runtime catalog writes or serving-time searches. At runtime, PrecomputePlan writes encoded summary payload bytes to the Summary Store and publishes partition/coverage/format/readiness/location as SummaryStateInstance metadata in the runtime inventory. QueryPlan resolves a ready instance from that inventory and reads its payload. So SDS is the contract across the definition catalog, installed plan bindings, runtime instance metadata, and payload store; the other metadata is not all in SummaryDefinition.
| the plan. The edges from both plans to that catalog are definition references | ||
| validated by catalog reads at installation, not runtime writes or serving-time | ||
| catalog searches. At runtime, PrecomputePlan writes summary payload bytes to | ||
| the store and publishes each instance's metadata to the inventory. QueryPlan |
There was a problem hiding this comment.
Better names for these might be -- SummaryPayloadStore and SummaryMetadataStore ?
| the store and publishes each instance's metadata to the inventory. QueryPlan | ||
| checks the inventory for a ready matching instance, then reads its payload from | ||
| the store. SDS describes this combined contract; its metadata is not all stored | ||
| in the Summary Catalog. Definition semantics live in the catalog, |
There was a problem hiding this comment.
This is confusing. Later, you describe the Catalog as storing SummaryDefinition, not metadata?
|
|
||
| query_plans: | ||
| q50: | ||
| read_state: &shared_read |
There was a problem hiding this comment.
Meaning of &shared_read and *shared_read ?
There was a problem hiding this comment.
Correction: those were YAML anchor/alias syntax and distracted from the design. I removed both and expanded the complete read_state binding under q50 and q99.
| | Object | Meaning | Changes when | | ||
| | --- | --- | --- | | ||
| | `SummaryDefinition` | Canonical input, operation, grouping, time semantics, algorithm and parameters | Summary semantics change | | ||
| | `SummaryStateInstance` | One stored partition, such as a series/pane or completed aggregate | Runtime publishes a new or replacement instance | |
There was a problem hiding this comment.
Does this include both payload and metadata?
There was a problem hiding this comment.
Correction: yes, conceptually. SummaryStateInstance means one complete logical SummaryStore entry: instance metadata plus its associated payload. An implementation may keep the bytes separately internally, but that does not create another architecture component.
| encoding: kll-binary-v1 | ||
| partition: {service: api, window_end: '12:05'} | ||
| coverage: {start_exclusive: '12:00', end_inclusive: '12:05'} | ||
| location: opaque-store-locator |
There was a problem hiding this comment.
what is this?
| encoding: kll-binary-v1 | ||
| partition_by: [service, window_end] | ||
|
|
||
| state_instances: |
There was a problem hiding this comment.
What does it mean for a plan to have state_instances? This YAML block seems to describe the metadata of a particular SDS instance.
There was a problem hiding this comment.
Correction: runtime instances are not part of the plan. The example now has separate installed_plan and runtime_summary_store sections. The latter is explicitly observed runtime data produced after the PrecomputePlan writes state.
| state_instances: | ||
| - id: state-api-1205 | ||
| plan_version: 42 | ||
| state_slot_id: latency-kll |
There was a problem hiding this comment.
what is slot?
There was a problem hiding this comment.
Correction: I renamed state_slot_id to stored_output_id and defined it as the plan-version-scoped binding ID for one persisted PrecomputePlan DAG output. The writer and QueryPlan readers use it to name the same output; it is not a memory slot or storage object.
| | `Desired` | Installed plan | The plan requires state for this slot and coverage | | ||
| | `Building` | Runtime inventory | Required state is being produced or recovered | | ||
| | `Ready` | Runtime inventory | Required schema and coverage are available | | ||
| | `Draining` | Runtime inventory | New work has stopped while existing use completes | | ||
| | `Retired` | Runtime inventory | New reads are prohibited; safe reclamation may follow | |
There was a problem hiding this comment.
What's the usecase for these?
There was a problem hiding this comment.
The five-phase table mixed plan intent, build progress, and storage retirement without a current SDS use case. I replaced it with the one decision the DAG path needs now: a QueryPlan read is eligible only when its bound instance exists, the payload is committed, and version, definition, format, partition, and coverage match. Building/draining/retirement remain existing runtime concerns rather than new SDS states.
# Conflicts: # docs/design_docs/README.md
Dependency stack: main → #768 → #737 → #749 → #771 → #728 → #770 → #763 → #765 → #761 → #742 → #759
Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).
Define how one Planner-selected computation becomes separate backend-executable PrecomputePlan and QueryPlan subgraphs joined through SDS state references.
Before this PR: The design mixed query readouts with maintenance under PrecomputePlan and described an extra catalog
Materializationobject. For example, two percentile queries sharing one KLL producer lacked a clear distinction between the one stored state writer and the two readouts.After this PR: PrecomputePlan owns construction and writes; QueryPlan owns bound state reads and readouts. The compiler binds both to one plan-version-scoped
stored_output_idand aSummaryDefinition. OneSummaryStoreowns two logical tables:summary_definitionsholdsSummaryDefinitionrows, andstored_summariesholds committedStoredSummaryrecords containing metadata and payload. The compiler supplies a definitions snapshot for installation.StoredOutputReferencenames the producer output;(plan_version, stored_output_id, population_key, window)locates a concrete record. QueryPlan checks coverage and format before reading it. This is a logical contract, not a proposal for separate metadata and payload services. The selected deployment guarantee and schedule/retention are carried from Planner; no new lifecycle model or standalone catalogMaterializationobject is introduced.The glossary, integration design, SDS architecture and migration plan describe identity, format/coverage validation, plan-version installation, fallback and migration boundaries. Worked examples show two queries sharing one producer and the consequence of losing its selected maintenance schedule.
Validation: Documentation diff, local link targets, and four YAML examples checked across five documents. This PR specifies the target contract; backend implementation is tracked separately.
Shared physical operator stack update
The shared runtime, operators and sketch codec are owned by ASAPPlanner #462, built on Planner #461. Backend #770 consumes them, #763 integrates ingestion and #765 integrates queries. The stack uses general semi-join and grouped Sort → Limit, with execution phase on the node. Independent library tests run in Planner; deployment acceptance stays in this backend stack.