Skip to content

Accelerated range aggregate returns an empty 200 at any instant not on the ingest grid #734

Description

@SheriffOreo

With a range aggregate provisioned, an instant query lands correctly only when it falls exactly on an ingest-grid point. At every other instant the accelerated path returns an empty result set with HTTP 200 and destination=asap, while Prometheus answers the identical query normally. Nothing in the reply signals that the answer is empty because of grid alignment rather than because there is no data.

This does not reproduce on ghcr.io/projectasap/asap-query-engine:v0.5.1, so it appears to be a regression — see below.

Environment

  • Engine and planner built from a fresh clone at main @ b3259449 using the repo's own Dockerfiles.
  • Backend prom/prometheus:v3.9.1, backend.type: prometheus, forward_unsupported_queries: true.
  • Data: 500 cells, metric m1, 1,024 stamps on a 900 s grid from 2025-01-01T00:00:00Z (epoch 1735689600). 1735862400 is an on-grid instant.
  • Provisioned workload includes sum_over_time(m1[24h]).

Steps to reproduce

T=1735862400                       # an on-grid instant
for off in 0 60 450 899 900; do
  curl -s "$ENGINE/api/v1/query" \
    --data-urlencode 'query=sum_over_time(m1[24h])' \
    --data-urlencode "time=$((T+off))" \
  | python3 -c "import sys,json
print(len(json.load(sys.stdin)['data']['result']))"
done

Expected

500 500 500 500 500 — Prometheus returns 500 series at all five offsets.

Actual

500 0 0 0 500

Extent

Sweeping all 901 one-second offsets across one 900 s grid step:

engine Prometheus
offsets returning 500 series 2 — exactly +0 s and +900 s 901 — all of them
offsets returning 0 series 899 — contiguous, +1 s through +899 s 0

So the served answer is empty at 899 of every 900 instants. The same happens for sum_over_time(m1[1h]) and quantile_over_time(0.99, m1[24h]), so it is not specific to one function.

Why this looks like a regression

Comparing like with like — both builds on quantile_over_time(0.99, m1[24h]), a shape both genuinely serve (destination=asap), same data, same Prometheus:

build +0 s +60 s +450 s +899 s +900 s
v0.5.1 500 500 500 500 500
main @ b3259449 500 0 0 0 500

Both are accelerating, so the difference is not "v0.5.1 just forwarded everything" — I checked destination= on each request. That suggests a bisect between the two rather than a design change.

Related

None of those record that the accelerated path returns an empty 200 rather than forwarding, which is the part that makes this silent.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions