Symptom
A production segment died mid-run with no usable diagnostic. It took a long time to even establish that the failure was in file I/O rather than in the solver.
Cause
Every MPI_FILE_OPEN call site passes ierr and then ignores it. On master:
src/simulation/m_start_up.fpp:311, 327, 343, 367, 444src/simulation/m_bubbles_EL.fpp:452, 470, 506, 547, 1859, 1900, 1916
and the same pattern in pre_process and post_process. When an open fails — a filesystem hiccup, a missing checkpoint, a quota or a metadata-server timeout, all routine at scale — execution continues with an invalid handle and the failure surfaces somewhere else entirely, or as a segfault.
Fix
Check ierr after each open and abort through s_mpi_abort naming the file and the MPI error string. Mechanical, but it converts a class of "the run died and we don't know why" into a one-line message.
Note
I do not have a reproducer — the original failure has not recurred — so this is filed on the code path rather than on a test case. The argument does not depend on the reproducer: an unchecked error code on a file open is worth closing on its own.
Found with Claude Code on OLCF Frontier.
Symptom
A production segment died mid-run with no usable diagnostic. It took a long time to even establish that the failure was in file I/O rather than in the solver.
Cause
Every
MPI_FILE_OPENcall site passesierrand then ignores it. On master:src/simulation/m_start_up.fpp:311, 327, 343, 367, 444src/simulation/m_bubbles_EL.fpp:452, 470, 506, 547, 1859, 1900, 1916and the same pattern in
pre_processandpost_process. When an open fails — a filesystem hiccup, a missing checkpoint, a quota or a metadata-server timeout, all routine at scale — execution continues with an invalid handle and the failure surfaces somewhere else entirely, or as a segfault.Fix
Check
ierrafter each open and abort throughs_mpi_abortnaming the file and the MPI error string. Mechanical, but it converts a class of "the run died and we don't know why" into a one-line message.Note
I do not have a reproducer — the original failure has not recurred — so this is filed on the code path rather than on a test case. The argument does not depend on the reproducer: an unchecked error code on a file open is worth closing on its own.
Found with Claude Code on OLCF Frontier.