[Design] Support batch query input http for PromQL

[zzylol]

Problem and why now

The network-control demo needs to evaluate roughly 10 PromQL queries together. Sending each query in a separate HTTP request adds client orchestration and request overhead. Elasticsearch's multi-search API is a useful precedent for submitting independent queries together.

Support an explicit batch of PromQL queries in one HTTP request while preserving existing single-query interfaces. The initial goal is convenient submission and independent results; shared query planning or computation is a separate optimization.

Intended users and MVP

• Network-control demo clients collecting several measurements for one control cycle.
• Applications that need several PromQL results at a common evaluation time.
• Proposed MVP: a batch of independent instant queries, with one shared evaluation timestamp and one result per input query.
• An expression such as rate(metric[5m]) can still be an instant query. Range-query batching is needed only if the caller needs evaluation across multiple timestamps, and should be included only if the demo requires it.

The API and semantics below are proposals for discussion, not approved design decisions.

Inspect existing HTTP APIs and redundancy first

Before selecting a new endpoint or implementing it, inspect the current HTTP API surface on the implementation base:

• Inventory existing query routes, methods, content types, protocol adapters, tenant-prefixed aliases, and any existing batch-capable interfaces.
• Compare instant/range GET and POST behavior, including JSON versus form parsing, parameter defaults, error formats, headers, and response metadata.
• Identify redundant endpoints and duplicated parsing, validation, dispatch, or response-formatting code. Distinguish required protocol compatibility aliases from accidental duplication.
• Check whether an existing API can express batching cleanly without changing established single-query behavior. Explain why a new public endpoint is needed if one is proposed.
• Inspect production and test router construction so endpoint coverage and behavior remain aligned.
• Record a concise route inventory and a recommendation: what to reuse, what duplication to consolidate within this change, and what broader cleanup should be tracked separately.

The purpose is a coherent API surface and shared behavior. Existing routes must not be removed or silently changed as incidental cleanup; any deprecation needs its own compatibility decision.

Starting points in the repository:

• data_plane/src/drivers/query/servers/http.rs
• data_plane/src/drivers/query/adapters/prometheus_http.rs
• data_plane/src/drivers/query/adapters/traits.rs

Inputs, outputs, and end-to-end outcome

Proposed request

Subject to the API review above, one possible interface is:

POST /api/v1/query_batch
Content-Type: application/json

{
  "time": 1790035200,
  "queries": [
    {
      "id": "traffic",
      "query": "sum(rate(network_receive_bytes_total[5m]))"
    },
    {
      "id": "errors",
      "query": "sum(rate(network_receive_errors_total[5m]))"
    }
  ]
}

• Each item contains a unique caller-supplied ID and a PromQL expression.
• A batch-level time applies to every item. If omitted, resolve the current time once for the batch.
• Reuse existing supported timestamp parsing where practical; document the accepted representation.
• Initially use the HTTP request's tenant and applicable execution headers for all items. Per-item overrides require an explicit need and precedence rules.

Proposed response

Return results in input order, echoing IDs and preserving each item's existing Prometheus response body. The proposed MVP omits per-item http_status until status propagation is defined consistently; see the fallback design below.

{
  "results": [
    {
      "id": "traffic",
      "response": {
        "status": "success",
        "data": {
          "resultType": "vector",
          "result": []
        }
      }
    },
    {
      "id": "errors",
      "response": {
        "status": "error",
        "errorType": "execution",
        "error": "Example execution failure"
      }
    }
  ]
}

The example illustrates the envelope, not an expected failure for the example expression. Preserve existing per-query warnings, infos, accuracy metadata, and source annotations when present.

End-to-end: the client submits its measurements once, receives a correlated outcome for every item, and can handle successful measurements and failures explicitly.

Simplest viable approach

1. Complete the existing-API review and choose the smallest compatible interface.
2. Parse and validate the batch envelope before starting query execution.
3. Resolve shared context and evaluation time once.
4. Execute each item through the existing single-query dispatch path, preserving routing, fallback, tenant context, engine selection, accuracy handling, and permitted forwarded headers.
5. Collect independent results into the batch response.

Reuse process_query_request or a narrowly extracted common helper as appropriate. Avoid internal HTTP calls back into this service and avoid building a second execution implementation.

Use bounded concurrency and a bounded batch size. Define request-body limits, deadline behavior, and cancellation before enabling concurrency; do not assume a parsed timeout is enforced by every execution path. Keep per-query metrics meaningful, with batch-level measurements only where useful.

Proposed design: compatibility with fallback databases

Batching is implemented at the ASAP HTTP boundary. A fallback database does not need to implement a batch endpoint: ASAP splits the batch into individual queries and reuses the existing single-query execution and fallback path.

Client -- one batch HTTP request --> ASAP
                                     |-- Q1 --> ASAP execution
                                     |-- Q2 --> fallback single-query API
                                     |-- Q3 --> fallback single-query API
                                     |-- ...
Client <-- ordered per-item results -- ASAP

This reduces client-to-ASAP requests; it does not necessarily reduce ASAP-to-database requests. Native backend batching can be considered later as an optimization, provided it preserves the same per-item contract.

Reuse existing dispatch and context

• Execute each item through process_query_request or its shared equivalent. Let the existing routing and fallback conditions determine where each query runs; do not introduce a separate batch-specific fallback policy.
• One item requiring fallback does not cause the entire batch to be forwarded. Mixed ASAP/fallback execution and all-fallback execution must both work.
• Resolve the shared evaluation timestamp once and pass it explicitly to every item, including every fallback request. Do not let downstream requests independently default to their own current time.
• Preserve the existing tenant, engine, and accuracy handling and the existing permitted-header forwarding rules. Distinguish ASAP execution context from headers actually forwarded to the backend; do not blindly copy all inbound headers.
• Reuse the configured fallback client and connection pool. For the Prometheus fallback, the existing client calls /api/v1/query with query, time, and optional timeout. If range batching is included, reuse the corresponding range path.
• When no fallback is configured, retain the existing per-query unsupported/error behavior.

Failure isolation, limits, and deadlines

• A backend query error, connection failure, malformed response, or timeout produces an error for that item without discarding other results.
• Bound outgoing fallback concurrency, accounting for simultaneous batch requests. Do not launch an unbounded number of backend requests.
• Specify how batch deadlines, item deadlines, queue time, and backend timeouts interact. The inspected Prometheus fallback currently uses a fixed 30-second HTTP timeout and also forwards an optional query timeout; reconcile these with the proposed deadline contract rather than assuming they already enforce it.
• Specify cancellation of queued and in-flight work when the batch deadline expires or the client disconnects. Do not assume dropping a local request guarantees that the remote database stops computation.

Response status limitation in the current implementation

In the inspected data_plane/src/drivers/query/fallback/prometheus.rs, the fallback reads the upstream HTTP status but returns FallbackResponse::Json(payload) without carrying that original status. Some transport failures are also converted into Prometheus-style JSON errors. Therefore, the batch layer cannot simply claim that a per-item http_status is the original backend HTTP status.

Proposed MVP: retain each item's status, errorType, error, and relevant response metadata, and omit per-item http_status. Normalize bodyless or non-JSON failures into a documented per-item error envelope so every accepted item still has a usable outcome. Preserve existing structured backend errors where available.

Alternative: extend the internal fallback result to carry status plus body, then explicitly define whether the exposed status represents the upstream response or ASAP's item outcome. Review all fallback implementations and test single-query compatibility before changing shared status handling. Include these inconsistencies in the existing HTTP API review rather than creating a second batch-only status policy.

Consistency boundary

A common evaluation timestamp does not guarantee that ASAP and the fallback database expose the same ingestion progress, data freshness, or storage snapshot. The proposed MVP provides independent results at a common logical evaluation time, not an atomic cross-backend read. If the controller requires a consistent snapshot or coordinated visibility, that is an additional design requirement to resolve before implementation.

Proposed behavior and constraints

• Independent failures: a PromQL validation or execution failure affects that item, while other items continue.
• Envelope errors: malformed JSON, an invalid shared timestamp, missing required envelope/item fields, empty batches, and duplicate IDs reject the request before execution. Document status codes for these and for exceeded limits.
• HTTP status: a valid, processed batch returns HTTP 200 with individual outcomes, even when some or all items fail. Whole-request failures retain appropriate non-2xx statuses.
• Correlation: return exactly one outcome per accepted item in input order.
• Time: all items use the same resolved evaluation timestamp. This does not imply an atomic storage snapshot or synchronized ingestion visibility.
• Completion: return one response after every item completes or reaches its applicable deadline. Define how queued items and client disconnects are handled.
• Compatibility: retain established single-query endpoints and wire formats. Document batching as an ASAP extension rather than a standard Prometheus API.
• Resource use: bound submitted work and concurrent execution, including the effect of multiple simultaneous batch requests.

Alternatives and quality attributes

• New explicit batch endpoint: clear request/response contract and minimal impact on single-query clients; adds another public route, so justify it against the API inventory.
• Extend an existing POST endpoint: fewer routes, but introduces multiple request and response shapes. Evaluate compatibility and maintainability before choosing.
• Client-side parallel requests: requires no new server API, but retains multiple HTTP requests and client orchestration.
• Automatically group separately arriving requests: introduces waiting-window and scheduling semantics; does not directly provide the requested one-request client interface.
• Combine expressions into one PromQL expression: generally does not preserve independent result identities, errors, types, and metadata.

Prioritize compatibility, predictable failure behavior, bounded resource use, and minimal implementation complexity. Do not promise execution speedups solely from HTTP batching.

Acceptance behavior

• Existing HTTP APIs have been inventoried; redundancy and the chosen reuse/consolidation strategy are documented.
• A client can submit approximately 10 supported PromQL queries in one HTTP request and receive one correlated result per query.
• At a fixed timestamp and against stable test data, each batch result matches the equivalent single-query result, including relevant metadata.
• Omitted evaluation time is resolved once and shared by all items.
• Mixed success/failure batches preserve successful results; an all-failure batch still returns per-item outcomes.
• Malformed envelopes, missing fields, duplicate IDs, empty batches, invalid shared time, and exceeded limits have documented, tested behavior.
• Response order is stable even when execution completion order differs.
• Tenant/header behavior and fallback dispatch match the applicable single-query behavior.
• Mixed ASAP/fallback and all-fallback batches work against a backend that only supports single-query requests; every forwarded item receives the shared timestamp and applicable context.
• Backend errors, connection failures, non-JSON responses, and timeouts produce correlated item errors while preserving other results.
• Missing-fallback behavior remains consistent with single-query execution.
• Tests verify the chosen response/status contract without assuming that original upstream HTTP status is currently preserved.
• Outgoing fallback concurrency and deadline behavior are verified, including queued work and simultaneous batches.
• Concurrency, deadlines, and cancellation have focused coverage appropriate to the chosen execution model.
• Existing single-query GET/POST behavior remains compatible.
• Production and test routers expose the intended batch route consistently.
• User-facing documentation includes a request/response example, limits, partial-failure handling, and the shared-time versus snapshot-consistency distinction.

Design questions to resolve

• Does the controller accept independent partial results, or require all measurements to succeed before acting?
• Is a common evaluation timestamp sufficient, or is a consistent storage snapshot required?
• Are instant queries sufficient for the demo, or are range-query batches required now?
• After reviewing the current APIs, should batching use a new route or extend an existing POST interface?
• What batch-size, concurrency, body-size, and deadline limits fit the expected workload?
• Which identified API redundancies should be addressed here, and which need separate cleanup issues?
• Should the MVP omit per-item HTTP status, or should shared fallback status propagation be fixed as part of this work?

Human decisions required

Pending maintainer input; no approvals or decisions recorded.