A client cannot tell whether an answer came from a sketch or from the backend, and for range queries the server log does not record it either. Every silent failure I hit while testing would have been visible immediately with one field naming the path that answered.
What an operator can see today
In the engine's stdout only:
destination=asap|prometheus — which path answered;asap_latency_ms — template match plus compute, when the query is served.
What an operator cannot see
1. Anything in the HTTP response. Served and forwarded replies are structurally identical — same keys, no header, no annotation:
served sum_over_time(m1[24h]) -> keys ['data','result','resultType','status']
forwarded sum_over_time(m1[23h]) -> keys ['data','result','resultType','status']
So a client cannot distinguish an approximate sketch answer from an exact backend one.
2. Range queries at all. An accelerated /api/v1/query_range emits no destination= line. Over 21 accelerated query_range calls the engine logged nothing; the same 21 calls against an engine with nothing provisioned logged 21 lines (http.rs, forwarded path). The accelerated range path therefore cannot be audited from the log, and I had to establish that it was being served indirectly, by noticing its answers differed from Prometheus's.
3. The cost of a forward. A forwarded 500-series reply logs total_latency_ms=0.30, which is not a network round trip — forwards are effectively untimed, so the log cannot be used to compare the two paths.
Suggestion
Either would have been enough for everything above:
- a response field or header naming the path that answered (and, if served, the aggregation used); or
- at minimum, emit the existing
destination= line for query_range as well as instant queries.
Context
Reproduced on main @ b3259449, engine and planner built from a fresh clone with the repo's own Dockerfiles, prom/prometheus:v3.9.1 backend, forward_unsupported_queries: true.
This is filed as an enhancement rather than a bug, but it is the reason two correctness problems I reported separately are silent to a client.
A client cannot tell whether an answer came from a sketch or from the backend, and for range queries the server log does not record it either. Every silent failure I hit while testing would have been visible immediately with one field naming the path that answered.
What an operator can see today
In the engine's stdout only:
destination=asap|prometheus— which path answered;asap_latency_ms— template match plus compute, when the query is served.What an operator cannot see
1. Anything in the HTTP response. Served and forwarded replies are structurally identical — same keys, no header, no annotation:
So a client cannot distinguish an approximate sketch answer from an exact backend one.
2. Range queries at all. An accelerated
/api/v1/query_rangeemits nodestination=line. Over 21 acceleratedquery_rangecalls the engine logged nothing; the same 21 calls against an engine with nothing provisioned logged 21 lines (http.rs, forwarded path). The accelerated range path therefore cannot be audited from the log, and I had to establish that it was being served indirectly, by noticing its answers differed from Prometheus's.3. The cost of a forward. A forwarded 500-series reply logs
total_latency_ms=0.30, which is not a network round trip — forwards are effectively untimed, so the log cannot be used to compare the two paths.Suggestion
Either would have been enough for everything above:
destination=line forquery_rangeas well as instant queries.Context
Reproduced on
main@b3259449, engine and planner built from a fresh clone with the repo's own Dockerfiles,prom/prometheus:v3.9.1backend,forward_unsupported_queries: true.This is filed as an enhancement rather than a bug, but it is the reason two correctness problems I reported separately are silent to a client.