Symptom
Memory access fault ... on address (nil) with no diagnostic, on a moving thin body at speed. A 4-cell-thick pitching panel died at the instant it crossed zero pitch at peak rate. The same case survived on 128 ranks and died on 256 and 512, which makes it look like a rank-count problem rather than an allocation one.
Cause
s_ibm_setup (src/simulation/m_ibm.fpp:126-131 on master):
call s_find_num_ghost_points(num_gps)
if (moving_immersed_boundary_flag) then
call s_mpi_allreduce_integer_sum(int(num_gps, 8), max_num_gps)
max_num_gps = min(max_num_gps*2_8, int(m + 1, 8)*int(n + 1, 8)*int(p + 1, 8))The ghost-point array is allocated identically on every rank, so its size has to hold the largest count any rank will reach as the body moves. What it is given is twice the global total at setup. That is generous per rank only while the body's ghost points are spread across ranks; once a thin body sweeps at speed through the few ranks it passes through, one rank's share can exceed a multiple of the setup-time global total. The more ranks there are, the smaller each rank's share at setup and the easier it is to exceed later — hence the false rank-count signature.
Nothing checks the count against the allocation on regeneration, so the overrun is a write off the end of a device array.
Fix
Two parts:
- Size it to twice the initial global total as now, but never below eight times the largest initial per-rank count, so the bound is set by the peak a single rank can reach rather than by an average.
- Check the count against the allocation on every regeneration in
s_update_mib and abort with the rank, the count and the size, instead of writing past it.
The second half is the part that matters even if the first heuristic is wrong: a device-side overrun with no diagnostic costs days, and an abort naming the numbers costs minutes.
Part of the moving-IB-across-ranks set filed today; see #1895, #1896, #1897, #1898, #1899.
Found with Claude Code on OLCF Frontier.
Symptom
Memory access fault ... on address (nil)with no diagnostic, on a moving thin body at speed. A 4-cell-thick pitching panel died at the instant it crossed zero pitch at peak rate. The same case survived on 128 ranks and died on 256 and 512, which makes it look like a rank-count problem rather than an allocation one.Cause
s_ibm_setup(src/simulation/m_ibm.fpp:126-131on master):The ghost-point array is allocated identically on every rank, so its size has to hold the largest count any rank will reach as the body moves. What it is given is twice the global total at setup. That is generous per rank only while the body's ghost points are spread across ranks; once a thin body sweeps at speed through the few ranks it passes through, one rank's share can exceed a multiple of the setup-time global total. The more ranks there are, the smaller each rank's share at setup and the easier it is to exceed later — hence the false rank-count signature.
Nothing checks the count against the allocation on regeneration, so the overrun is a write off the end of a device array.
Fix
Two parts:
s_update_miband abort with the rank, the count and the size, instead of writing past it.The second half is the part that matters even if the first heuristic is wrong: a device-side overrun with no diagnostic costs days, and an abort naming the numbers costs minutes.
Part of the moving-IB-across-ranks set filed today; see #1895, #1896, #1897, #1898, #1899.
Found with Claude Code on OLCF Frontier.