Version
v24.11.1 (V8 13.6.233.10-node.28)
Platform
Linux x86_64, kernel 6.1.134-152.225.amzn2023.x86_64
libc: musl (Alpine-based container image)
cgroup v2, CPU quota = 12 cores, memory limit 24576 MiB, swap off
Subsystem
deps/v8 (compiler)
What steps will reproduce the bug?
I don't have a minimal reproduction, and I want to be upfront about that.
The workload is a Vitest 5 run in forks pool with isolate: true: one short-lived node child process per test file, 1579 children per run, 32 concurrent against a 12-core quota, ~740 s wall clock. Each child spends most of its life loading and evaluating modules (transform 61%, import 27%, tests 5%), so it is optimizing-compiler-heavy startup, repeated 1579 times. The children are plain JS workloads — component/unit tests on jsdom. No node:http2, no network server, no HTTP/2 client anywhere in them.
Occasionally one child dies on a signal. The rest of the run is green.
How often does it reproduce? Is there a required condition?
7 runs with at least one crash out of 87 runs (~8%), i.e. roughly 1 in 20000 child processes. One run produced three separate events.
CPU oversubscription appears to be required: I have never reproduced it on an idle machine, and a fork-storm stress test (tens of thousands of bare node -e '' spawns) never hit it — the child has to actually be compiling JavaScript. Anyone trying to reproduce should think in terms of total contention (host baseline + concurrent children ÷ available cores); the loadavg in my logs is sampled at run start and excludes the load the run itself creates.
What is the expected behavior?
The child process should not die.
What do you see instead?
Three signatures so far, all in the same subsystem:
- A.
Check failed: (data_) != nullptr. with SIGTRAP
- B.
Check failed: IsJSFunction(). with SIGTRAP
- C. plain SIGSEGV with no V8 output at all
Signature A verbatim; B is identical in shape:
#
# Fatal error in , line 0
# Check failed: (data_) != nullptr.
#
#
#
#FailureMessage Object: 0x7fa7fa0b27e0
----- Native stack trace -----
The parent reports Worker exited unexpectedly with signal <SIG>. SIGTRAP rather than SIGABRT is consistent with --hard-abort (default true).
At 13.6.233.10 both CHECKs sit in src/compiler/heap-refs.h: A is CHECK_NOT_NULL(data_), which appears at 360 and 386 and is reached through the latter in a release build, and B is the CHECK(Is##Name()) in DEFINE_REF_CONSTRUCTOR. The null check runs first, so B means a non-null ObjectData of the wrong type.
Additional information
What I'm asking for. Please consider backporting these to v24.x:
All three are already-accepted upstream fixes for unsynchronized reads in JSHeapBroker, which is exactly where we crash; the CLs carry the mechanism. Signature B — non-null, wrong type — matches both shapes they fix: a slot holding either a JSFunction or a Map read mid-flip (b1c2cda9b), and a map observed before Factory::InitializeMap's stores are visible (0bb186fd9).
Version containment, checked with gh api repos/v8/v8/compare/<tag>...<sha>:
| Node branch |
V8 |
has c3553509f / b1c2cda9b |
has 0bb186fd9 |
| v24.x |
13.6.233.17 |
no |
no |
| v25.x |
14.1.146.11 |
no |
no |
| v26.x / main |
14.6.202.34 |
yes |
no |
Staying on 24.x doesn't help: 13.6.233.10 → 13.6.233.17 adds 5 substantive cherry-picks, none touching heap-refs, the broker, or handles.
I'm not asking anyone to debug my workload — the report is that a supported LTS line ships a V8 predating this cluster of fixes. Cost disclosed up front: b1c2cda9b widens a public return type to OptionalObjectRef and updates four caller files that have moved a lot since 13.6, so it isn't a mechanical cherry-pick. "Too invasive for an LTS V8" is a legitimate answer; I'd then suggest the class is still worth recording against 24.x, since the failure mode is a hard process death with no diagnosable output.
What distinguishes this from #64841. Same shape — intermittent Linux crash, JSHeapBroker involved, contention required, --no-maglev making it go away — but that one resolved to invalid nghttp2 accesses and is now scoped to HTTP/2, which this workload never touches, so I'm deliberately not commenting there.
The distinguishing evidence is that for some of our crashes there was no third-party native code in the process at all. I hooked process.dlopen from a --require shim inherited by the forked children and enumerated a full run: the children load at most bufferutil (884 of 1579) and bigint-buffer (18), while the bundler's native binding stays in the parent because children receive transformed modules over IPC. Of the six distinct test files that have crashed, two ran in processes with no addon at all, and the most recent crash came from a run where bufferutil and utf-8-validate were blocked at require() time. Node's own built-in native code remains in scope — I can't turn that off to test it.
Ruled out, two observability notes, and the mitigation in flight
Ruled out. Container OOM: cgroup memory.peak 10844/24576 MiB, memory.events oom_kill=0, swap off, pids 385 of 37827 — and it would be SIGKILL, with a V8 heap OOM being SIGABRT. musl's 128 KiB default thread stack: concurrent compilation runs on Node's platform workers, which node_platform.cc creates via uv_thread_create → libuv's uv__thread_stack_size() (RLIMIT_STACK, else 2 MB), so the musl default never applies; verified in node:24.11.1-alpine, where V8's own regress-crbug-388320179.js passes a 2000-link prototype chain that a 128 KiB stack would have killed at ~340. Stale V8 code cache: the runner deletes NODE_COMPILE_CACHE for the children and nothing sets it.
Observability, independent of the root cause. The release V8_Fatal path passes an empty file and line 0, so the assertion text is the only usable signal — unique for both of my signatures, but it would not be for a commoner CHECK. And ----- Native stack trace ----- prints its header and then nothing on musl, because Node's backtrace depends on glibc's execinfo.h; a reader cannot distinguish "no frames available on this libc" from "the process died before it could walk the stack". A one-line explanation instead of an empty section would save people real time, which also bears on the Alpine tier-2 discussion in #62764.
Mitigation in flight. --no-concurrent-recompilation for the children, both as mitigation and as the only discriminating experiment I have left: a recurrence under that flag would falsify the attribution above in a single run. I'll report back either way, though a clean result is much weaker evidence — at an 8% per-run base rate it takes ~28 consecutive clean runs to reach p≈0.1.
Body condensed at maintainer request; the original long-form version remains in this issue's edit history.
Version
v24.11.1 (V8
13.6.233.10-node.28)Platform
Subsystem
deps/v8 (compiler)
What steps will reproduce the bug?
I don't have a minimal reproduction, and I want to be upfront about that.
The workload is a Vitest 5 run in
forkspool withisolate: true: one short-livednodechild process per test file, 1579 children per run, 32 concurrent against a 12-core quota, ~740 s wall clock. Each child spends most of its life loading and evaluating modules (transform 61%, import 27%, tests 5%), so it is optimizing-compiler-heavy startup, repeated 1579 times. The children are plain JS workloads — component/unit tests on jsdom. Nonode:http2, no network server, no HTTP/2 client anywhere in them.Occasionally one child dies on a signal. The rest of the run is green.
How often does it reproduce? Is there a required condition?
7 runs with at least one crash out of 87 runs (~8%), i.e. roughly 1 in 20000 child processes. One run produced three separate events.
CPU oversubscription appears to be required: I have never reproduced it on an idle machine, and a fork-storm stress test (tens of thousands of bare
node -e ''spawns) never hit it — the child has to actually be compiling JavaScript. Anyone trying to reproduce should think in terms of total contention (host baseline + concurrent children ÷ available cores); theloadavgin my logs is sampled at run start and excludes the load the run itself creates.What is the expected behavior?
The child process should not die.
What do you see instead?
Three signatures so far, all in the same subsystem:
Check failed: (data_) != nullptr.with SIGTRAPCheck failed: IsJSFunction().with SIGTRAPSignature A verbatim; B is identical in shape:
The parent reports
Worker exited unexpectedly with signal <SIG>. SIGTRAP rather than SIGABRT is consistent with--hard-abort(default true).At
13.6.233.10both CHECKs sit insrc/compiler/heap-refs.h: A isCHECK_NOT_NULL(data_), which appears at 360 and 386 and is reached through the latter in a release build, and B is theCHECK(Is##Name())inDEFINE_REF_CONSTRUCTOR. The null check runs first, so B means a non-nullObjectDataof the wrong type.Additional information
What I'm asking for. Please consider backporting these to
v24.x:c3553509f[compiler] Thread-safe IsArrayOrObjectPrototypeb1c2cda9b[compiler] Thread-safe MapRef::GetConstructor,GetBackpointer0bb186fd9[objects][compiler] Release-store the initial map into the prototype tupleAll three are already-accepted upstream fixes for unsynchronized reads in
JSHeapBroker, which is exactly where we crash; the CLs carry the mechanism. Signature B — non-null, wrong type — matches both shapes they fix: a slot holding either aJSFunctionor aMapread mid-flip (b1c2cda9b), and a map observed beforeFactory::InitializeMap's stores are visible (0bb186fd9).Version containment, checked with
gh api repos/v8/v8/compare/<tag>...<sha>:c3553509f/b1c2cda9b0bb186fd9Staying on 24.x doesn't help:
13.6.233.10 → 13.6.233.17adds 5 substantive cherry-picks, none touchingheap-refs, the broker, or handles.I'm not asking anyone to debug my workload — the report is that a supported LTS line ships a V8 predating this cluster of fixes. Cost disclosed up front:
b1c2cda9bwidens a public return type toOptionalObjectRefand updates four caller files that have moved a lot since 13.6, so it isn't a mechanical cherry-pick. "Too invasive for an LTS V8" is a legitimate answer; I'd then suggest the class is still worth recording against 24.x, since the failure mode is a hard process death with no diagnosable output.What distinguishes this from #64841. Same shape — intermittent Linux crash,
JSHeapBrokerinvolved, contention required,--no-maglevmaking it go away — but that one resolved to invalid nghttp2 accesses and is now scoped to HTTP/2, which this workload never touches, so I'm deliberately not commenting there.The distinguishing evidence is that for some of our crashes there was no third-party native code in the process at all. I hooked
process.dlopenfrom a--requireshim inherited by the forked children and enumerated a full run: the children load at mostbufferutil(884 of 1579) andbigint-buffer(18), while the bundler's native binding stays in the parent because children receive transformed modules over IPC. Of the six distinct test files that have crashed, two ran in processes with no addon at all, and the most recent crash came from a run wherebufferutilandutf-8-validatewere blocked atrequire()time. Node's own built-in native code remains in scope — I can't turn that off to test it.Ruled out, two observability notes, and the mitigation in flight
Ruled out. Container OOM: cgroup
memory.peak10844/24576 MiB,memory.eventsoom_kill=0, swap off,pids385 of 37827 — and it would be SIGKILL, with a V8 heap OOM being SIGABRT. musl's 128 KiB default thread stack: concurrent compilation runs on Node's platform workers, whichnode_platform.cccreates viauv_thread_create→ libuv'suv__thread_stack_size()(RLIMIT_STACK, else 2 MB), so the musl default never applies; verified innode:24.11.1-alpine, where V8's ownregress-crbug-388320179.jspasses a 2000-link prototype chain that a 128 KiB stack would have killed at ~340. Stale V8 code cache: the runner deletesNODE_COMPILE_CACHEfor the children and nothing sets it.Observability, independent of the root cause. The release
V8_Fatalpath passes an empty file and line 0, so the assertion text is the only usable signal — unique for both of my signatures, but it would not be for a commonerCHECK. And----- Native stack trace -----prints its header and then nothing on musl, because Node's backtrace depends on glibc'sexecinfo.h; a reader cannot distinguish "no frames available on this libc" from "the process died before it could walk the stack". A one-line explanation instead of an empty section would save people real time, which also bears on the Alpine tier-2 discussion in #62764.Mitigation in flight.
--no-concurrent-recompilationfor the children, both as mitigation and as the only discriminating experiment I have left: a recurrence under that flag would falsify the attribution above in a single run. I'll report back either way, though a clean result is much weaker evidence — at an 8% per-run base rate it takes ~28 consecutive clean runs to reach p≈0.1.Body condensed at maintainer request; the original long-form version remains in this issue's edit history.