Check native compiler flags against per-CPU-target references before building

[casparvl]

Depends on #312 . That should be merged first, then the diff between this and main should be (a bit) smaller.

Check native compiler flags against per-CPU-target references before building

The problem

Every CPU target in EESSI (e.g. x86_64/intel/icelake) is built on a host of that type, with -march=native (-mcpu=native on aarch64). Until now there has been a single build host per CPU target, so all binaries in a given CPU prefix were optimised for exactly the same set of instruction-set extensions.

If we want redundancy (i.e. multiple build hosts for the same CPU target, possibly at different sites), we need to ensure that all systems that build for the "same" CPU type expose the same CPU features. In the past, we have seen that this is not always the case: e.g. we have seen different flags being reported by lscpu on different zen5 issues, and even had binary compatibility issues between binaries built on Grace Hopper nodes at SURF ETP vs Jureca. The fundamental issue is that if two build hosts for the same CPU target translate -march=native differently, the result is an inconsistent software stack in a single CPU prefix, where some binaries don't run on every system of that CPU target.

The solution in this PR

Early in EESSI-install-software.sh (directly after the software subdirectory is verified, before anything is installed), we now check that, on the current build host, the compilers translate the native architecture flag into exactly the same target flags as recorded in a reference for that CPU target. The build stops if they don't.

Rather than comparing CPU flags from lscpu or a library like cpu_features (which would only be a surrogate), we ask the compiler driver itself what it resolves the native flag into, since that is what determines the generated code:

• GCC: gcc <native flag> -### -E - < /dev/null. All -m* options passed to cc1 are compared (-march/-mcpu, -mtune, and the full list of -m<ext>/-mno-<ext>). --param options such as l1-cache-size are ignored, since they only affect tuning.
• LLVM: clang <native flag> -### -c -x c - -o /dev/null. The -target-cpu, -tune-cpu, -target-abi and -target-feature options passed to clang -cc1 are compared.

The native flag is the same one EasyBuild uses (COMPILER_OPTIMAL_ARCHITECTURE_OPTION):
-march=native on x86_64, and -mcpu=native on aarch64 for both GCC and LLVM.

Since the result depends on the compiler version (e.g. on Neoverse-N1, GCC 12.x gives -mcpu=ares+crypto+ssbs+noprofile while GCC 13.2 gives -mcpu=ares+crc+crypto+ssbs), references are keyed on CPU target, compiler and compiler version: scripts/native_flags/references/<EESSI_SOFTWARE_SUBDIR>/<gcc|clang>-<version>.txt

The compilers to check are taken from the toolchains supported in the EESSI version being built:

• Supported toolchains: scripts/native_flags/get_supported_toolchains.py reads the supported toplevel toolchains for the EESSI version from eessi_supported_toolchains.json (introduced in Move supported toplevel toolchains from eb_hooks.py into a separate JSON for easier define-once-and-reuse #312 , and also used by eb_hooks.py). It includes toolchains regardless of their min_easybuild_version, since that only says which EasyBuild version can install them, and adds site toolchains from
  $EESSI_SITE_TOP_LEVEL_TOOLCHAINS_<version>.
• Compilers per toolchain: scripts/native_flags/check_native_flags.sh loads each toolchain module in a subshell. It takes gcc and clang from $PATH, keeping only binaries that come from a module installation (an $EBROOT* prefix) and dropping duplicates (e.g. GCC 14.3.0 from both foss/2025b and lfoss/2025b). A toolchain whose module is unavailable, or deliberately fails to load (e.g. foss/2022b on zen4 in 2023.06), is skipped with a warning.
• Generating references: check_native_flags.sh --generate writes the references for the current host.

Behaviour and overrides:

• A mismatch is a fatal error, and a diff (reference vs. this host) is printed.
• A missing reference is also an error by default. Set $EESSI_NATIVE_FLAGS_ALLOW_MISSING_REFERENCE to make it a warning.
• $EESSI_SKIP_NATIVE_FLAGS_CHECK skips the check entirely. It is always skipped for --generic builds.
• $EESSI_NATIVE_FLAGS_REFERENCE_DIR overrides the reference location.

Included references

References were generated on the current (AWS) build cluster, so that any future redundant build host must match what the existing binaries were built with. They cover EESSI 2023.06, 2025.06 and 2026.06 (GCC 12.2.0 to 15.2.0, Clang 20.1.8 and 21.1.8) for:

• aarch64/{neoverse_n1,neoverse_v1,aws/graviton4}, x86_64/amd/{zen2,zen3,zen4},
• x86_64/intel/{haswell,skylake_avx512,cascadelake,icelake,sapphirerapids}

Limitations

• CPU targets not built on that cluster have no references yet: x86_64/amd/zen5, x86_64/intel/graniterapids, aarch64/nvidia/grace, aarch64/google/axion, aarch64/a64fx. Since a missing reference is an error by default, builds for these targets will fail until someone runs scripts/native_flags/check_native_flags.sh --generate on their current build host (inside the EESSI environment for each EESSI version) and adds the output. Alternatively, set EESSI_NATIVE_FLAGS_ALLOW_MISSING_REFERENCE on those build hosts for now.
• New compiler versions need new references. When a new toolchain is added to an EESSI version, the first build after its compiler is deployed will fail until references for that compiler version are added.
• Kernel version can cause mismatches that are (probably) harmless. On aarch64, GCC and LLVM derive the CPU features from the hwcaps the kernel reports in /proc/cpuinfo. All current build nodes run an EL8 kernel (4.18), which does not report pointer authentication (paca/pacg), so GCC resolves to e.g. -mcpu=neoverse-v1+…+nopauth. A redundant build host with a newer kernel (e.g. EL9, 5.14) will likely report pointer authentication, drop +nopauth, and fail the check for neoverse_v1 and aws/graviton4. Strictly speaking that is correct, since the flags really differ. In practice the difference is probably harmless: GCC only emits pointer-authentication instructions with -mbranch-protection, and those instructions execute as no-ops on hardware without the feature. We'll need to decide either to keep build hosts on the same kernel generation, or to allow known-harmless differences.
• The Clang check is weaker on aarch64. For the same hosts, Clang reports +pauth (and +mte, +fpac on Graviton4) even though the kernel does not report those features. Clang appears to start from the default features of the core (identified from the CPU model) and add what /proc/cpuinfo reports, rather than removing defaults the kernel doesn't report (not verified in the LLVM source). If so, a host whose kernel fails to report a feature that is a default for its core would not change Clang's output. GCC's would, so on aarch64 the GCC check is the more reliable one. On x86_64 both read CPUID and look sound.
• The compat-layer GCC is not checked.
• LLVM installations that aren't a toolchain compiler (e.g. LLVM/20.1.7-GCCcore-14.2.0, which also ships clang) are not checked.
• Software that does its own CPU detection at build time (e.g. GROMACS SIMD detection, rustc -C target-cpu=native with rustc's bundled LLVM) is not covered.
• rompi/2025a is not installed on any of the targets covered here, so it has no references. A reference for ROCm clang would also be stored as clang-<version> and could clash with an upstream LLVM version.
• Runtime: the check loads every toolchain module, which adds up to about 20 seconds per build (mostly for 2025.06).
• No CI tests are included for the new scripts.

AI disclosure

This PR was developed together with an AI coding assistant (Claude, via Claude Code) in an interactive session that I steered and checked throughout. Concretely:

• Approach: the approach was mine: I proposed asking GCC directly via gcc -march=native -### -E - rather than using lscpu/cpu_features, and keying the references on CPU model and compiler version. I asked the assistant to review the approach, find an LLVM equivalent, and implement a loop over all GCC and LLVM compilers supported in the EESSI version being built.

• Initial implementation: the assistant read the build scripts and eb_hooks.py, used EESSI_SUPPORTED_TOP_LEVEL_TOOLCHAINS as the list of supported toolchains, and checked in the EasyBuild source that EasyBuild uses -mcpu=native (not -march=native) on aarch64. It wrote the two scripts and the hook. It tested them on an aarch64 (Neoverse-N1) login node against EESSI 2023.06, 2025.06 and 2026.06, covering reference generation, an exact match, deliberately edited references for GCC and Clang (both detected, with a readable diff), and the missing-reference path.

• Error by default: at my request, it made a missing reference an error by default, with an environment variable to turn it into a warning.

• References on the build cluster: I then asked it to generate references on our Slurm build cluster, one partition per CPU target. It ran check_native_flags.sh --generate in a batch job on each non-generic partition, for all three EESSI versions, and recorded lscpu, /proc/cpuinfo flags, kernel version and archdetect output alongside. Problems it found and fixed along the way:

• foss/2022b deliberately fails to load on Zen4, which initially made the check fail. It changed module-load failures to a warning.

• Cross-check against /proc/cpuinfo: I asked whether the GCC output matched the CPU flags from lscpu//proc/cpuinfo. On all x86_64 targets, every extension GCC enables is reported by the kernel, and none it disables is (apart from naming differences such as sse3/pni). On aarch64 the GCC output matches the kernel's Features. This comparison surfaced the +nopauth and Clang-on-aarch64 limitations described above.

• Writing: the assistant wrote the code, the commit message and a draft of this description. I have reviewed the code & commit message in a draft PR and make final changes myself where I think they're needed before marking the PR as ready for review.

  🤖 Generated with Claude Code

[casparvl]

TODO's: make sure that we add references for all the existing build architectures, before we merge this PR.