Skip to content

[Metrics SDK] Fix spatial re-aggregation for asynchronous instruments - #4635

Merged
marcalff merged 11 commits into
open-telemetry:mainfrom
nikhilbhatia08:1724-reaggregation-fix-for-dimensions-dropping
Oct 1, 2026
Merged

marcalff merged 11 commits into
open-telemetry:mainfrom
nikhilbhatia08:1724-reaggregation-fix-for-dimensions-dropping

Conversation

@nikhilbhatia08

Copy link
Copy Markdown
Contributor

Fixes #1724

Changes

Fix spatial re-aggregation for asynchronous instruments. A view which drops attributes is now applied to asynchronous instruments as well, and the observations which collapse onto the same attribute set are re-aggregated (summed up for the additive instruments) instead of the last observation overwriting the previous ones. Note that AsyncMetricStorage's constructor now takes the view's AttributesProcessor, matching SyncMetricStorage.

For significant contributions please make sure you have completed the following items:

  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

@nikhilbhatia08
nikhilbhatia08 requested a review from a team as a code owner September 23, 2026 20:14
@nikhilbhatia08 nikhilbhatia08 changed the title Fix spatial re-aggregation for asynchronous instruments [Metrics SDK] Fix spatial re-aggregation for asynchronous instruments Sep 23, 2026
@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.43750% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 86.70%. Comparing base (754928d) to head (c208050).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...telemetry/sdk/metrics/state/async_metric_storage.h 98.42% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4635      +/-   ##
==========================================
+ Coverage   86.66%   86.70%   +0.05%     
==========================================
  Files         525      525              
  Lines       20481    20596     +115     
==========================================
+ Hits        17748    17856     +108     
- Misses       2733     2740       +7     
Files with missing lines Coverage Δ
sdk/src/metrics/meter.cc 81.36% <100.00%> (ø)
...telemetry/sdk/metrics/state/async_metric_storage.h 95.75% <98.42%> (+2.57%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lalitb

lalitb commented Sep 24, 2026

Copy link
Copy Markdown
Member

@nikhilbhatia08 - Thanks for working on this. The review uncovered some issues in the existing async storage design that make this broader than the original fix. Could you hold off on further implementation while I work through the design? Your PR is a useful starting point, and we can split the implementation once the approach is clear.

@nikhilbhatia08

Copy link
Copy Markdown
Contributor Author

Yeah Sure @lalitb, that works, Thanks!

@lalitb

lalitb commented Sep 24, 2026

Copy link
Copy Markdown
Member

@nikhilbhatia08 - Could we calculate deltas before removing attributes for monotonic Sum aggregation?

Suppose the view removes the attribute named version:

First collection:
  Observe(20, {version: "v1"})
  Observe(10, {version: "v2"})

Next collection:
  Observe(25, {version: "v1"})
  // No observation for version "v2"

Removing version first combines the two original series into one output group. Its total changes from 30 to 25, giving 25 - 30 = -5. But the series with version="v1" actually increased by 25 - 20 = 5.

Could we:

  1. Look up the previous value using the original attributes, including version.
  2. Calculate that series' delta and update its previous value.
  3. Apply the view, then merge the delta into the matching output group.

The existing delta_hash_map_ stays around until Collect(), so it can also combine contributions from different callbacks. If the first two observations above come from separate callbacks, the current code can report -10 instead of their combined value of 30.

With the existing cumulative reconstruction, this example produces 30 -> 35, retaining the earlier contribution of 10 from version="v2". Let's preserve that behavior for this fix and add a test covering it. We can discuss whether cumulative output should use snapshot semantics separately.

Non-monotonic sums, such as UpDownCounters, need different handling. If two processes use 100 and 50 units of memory, and the second process exits, the current total should become 100. Could we sum their filtered absolute observations across all callbacks first, then calculate the difference once in Collect()? Reset that accumulator for each collection round so it contains only that round's observations.

Please add tests for the two-callback case, the disappearing counter, and the UpDownCounter example. For output overflow, check that contributions combine correctly in the first collection, including in the otel.metric.overflow point. Baseline saturation in later collections is an existing limitation related to #2350.

If a filtered group disappears entirely, #4484 is addressing that stale-output behavior through #4108. Please keep this change compatible with its baseline-preservation and stale-suppression handling.

@lalitb

lalitb commented Sep 25, 2026

Copy link
Copy Markdown
Member

@nikhilbhatia08 - Thanks, the change look good. can you fix the CI so it is in good state?

@linux-foundation-easycla

linux-foundation-easycla Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

- Add the includes include-what-you-use asks for in async_metric_storage_test.cc
  (<map>, attribute_utils.h, aggregation_config.h) and use <cstddef> instead of
  the deprecated <stddef.h>, matching the rest of sdk/test/metrics.
- Drop noexcept from the new private helpers (RecordMonotonicSum, AccumulateDelta,
  FilterAttributes). They allocate and can throw; the noexcept boundary stays on
  the public Record/Collect entry points, so behaviour is unchanged and
  bugprone-exception-escape no longer fires on them.
@nikhilbhatia08
nikhilbhatia08 force-pushed the 1724-reaggregation-fix-for-dimensions-dropping branch from 7087038 to 74c21dd Compare September 25, 2026 17:02
@nikhilbhatia08

Copy link
Copy Markdown
Contributor Author

Hey @lalitb fixed the ci, its green now, thanks!

@lalitb lalitb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks.

@marcalff
marcalff merged commit 6b14d5d into open-telemetry:main Oct 1, 2026
77 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Metrics SDK] Async aggregation doesn't do re-aggregation properly when the spatial dimensions are dropped

3 participants