Compute the attribution pc with integer arithmetic

[rkennke]

What does this PR do?:

Makes attributionPC() subtract 1 from the return address with uintptr_t arithmetic instead of const char* arithmetic.

Motivation:

During an optimistic unwind, walkVM can read garbage such as 0 or 0x1 from a return-address slot and hand it to resolveNativeFrameForWalkVM() → attributionPC(). Offsetting a pointer to or from null is undefined behavior, and UBSan's pointer-overflow check reports it:

• Nightly Sanitized Run on main (2026-09-30 onward): stackWalker.inline.h:64:65: runtime error: applying non-zero offset 18446744073709551615 to null pointer (pc == nullptr), about 30 reports per run.
• feat(wall): support signal suppression for unfiltered threads #663 CI, glibc-aarch64/asan/JDK 25: applying non-zero offset to non-null pointer 0x000000000001 produced null pointer (pc == 0x1), from both the wall-clock and CPU signal handlers.

In the sanitized test config (halt_on_error=0, log_path=/tmp/asan.log) these don't fail a test. Instead the test JVM exits with code 1 at shutdown, and the report only appears in the asan-logs-* artifact.

Unsigned wrap-around is well defined, and the resulting address (UINTPTR_MAX or 0) still matches no library and no FDE, so attribution output is unchanged. This also covers the 0x1 case, which a pc != nullptr guard does not.

I checked the other address arithmetic added by the return-address attribution changes (#786, #813, #830, #831). The remaining sites already use integer arithmetic (pc() -= 1 on uintptr_t&, pc_offset on uintptr_t) or operate on addresses already validated to lie inside a library (the DW_PC_OFFSET recovered pc in dwarfStep.inline.h), so attributionPC() is the only place that needed changing.

How to test the change?:

New gtest ReturnAddressAttributionCharacterizationTest.AttributionPcOfGarbageNearNullIsDefined feeds 0 and 0x1 through attributionPC().

• gtestAsan_returnAddressAttribution_ut (glibc aarch64 container): aborts with the UBSan report without the fix and passes with it.
• gtestDebug_returnAddressAttribution_ut (macOS): passes.

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a security review.
• This PR doesn't touch any of that.
• JIRA: [PROF-XXXX]

🤖 Generated with Claude Code

[datadog-datadog-prod-us1]

More details

Near-null return addresses now wrap through uintptr_t arithmetic without pointer-overflow undefined behavior while retaining the same attribution outcome.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 17db8d9 · @DataDog review to ask questions}

[dd-octo-sts]

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 17db8d91

[dd-octo-sts]

CI Test Results

Run: #37041712180 | Commit: 231199d | Duration: 1h 45m 22s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK	glibc-aarch64/debug	glibc-amd64/debug	musl-aarch64/debug	musl-amd64/debug
8	-	✅	-	-
8-ibm	-	✅	-	-
8-j9	✅	✅	-	-
8-librca	-	-	✅	✅
8-orcl	-	✅	-	-
11	-	✅	-	-
11-j9	✅	✅	-	-
11-librca	-	-	✅	✅
17	✅	✅	-	-
17-graal	✅	✅	-	-
17-j9	✅	✅	-	-
17-librca	-	-	✅	✅
21	✅	✅	-	-
21-graal	✅	✅	-	-
21-librca	-	-	✅	✅
25	✅	✅	-	-
25-graal	✅	✅	-	-
25-librca	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0

---

Updated: 2026-10-03 07:04:29 UTC