ENH: Replace the shell wheel build system with Python (supersedes #302)

[hjmjohnson]

Replaces the shell and PowerShell wheel-build system with a pixi-driven
Python one, and moves ITK 6 to stable-ABI (abi3) wheels on
manylinux_2_28, macOS arm64 14.0, and Windows x86_64. Supersedes #302.

Status: verified on all four platforms. Draft pending maintainer review.

One item needs a decision before merge: the rebuilt macOS cache is
verified but not yet published to ITKPythonBuilds. The two Linux caches
and the Windows cache are published.

Platform verification — clean builds from this branch

Every row is a from-scratch build of this branch, not an incremental one.

Platform	Result	Time	Wheels
Linux x86_64	pass	—	7 × cp311-abi3-manylinux_2_28_x86_64
Linux aarch64	pass on retry	39 min	7 × cp311-abi3-manylinux_2_28_aarch64
macOS arm64	pass	93 min	7 × cp311-abi3-macosx_14_0_arm64
Windows x86_64	pass	—	7 × cp311-abi3-win_amd64

macOS binaries were checked with otool -l: real .so files report
minos 14.0, matching their macosx_14_0_arm64 tag. That check matters
because a component wheel tagged below its true minos installs on older
macOS and then fails in dyld at import — installable and broken, rather
than loudly uninstallable.

Remote-module end-to-end (the consumer path)

Proven on Windows x86_64 and Linux x86_64, each run as a fresh
consumer with zero ITK wheels present beforehand, so a leftover
artifact could not produce a false pass.

Linux x86_64, ITKBioCell, 8 minutes, rc=0:

• itk_biocell-1.0.0-cp311-abi3-manylinux_2_28_x86_64.whl, 0.58 MB
• Requires-Dist: itk<7,>=6.0.0b1 — the dependency rewrite fired
• 2 bundled libraries — genuinely repaired by auditwheel
• pyproject.toml restored byte-identical after the build

An earlier Windows pass was found to be riding on leftover wheels; the
zero-wheel precondition was added in response and both platforms were
re-run under it.

Unit tests and hooks

• pytest: 97 passed. pytest is a developer-only dependency; the
  normal build path does not import it.
• pre-commit run --all-files: exit 0, every hook Passed
  (black, ruff, shellcheck, shfmt, taplo, yaml/toml, whitespace).

Known caveat: intermittent aarch64 link truncation

One clean aarch64 build failed while linking:

FAILED: lib/libITKIOStimulate-6.0.a
ranlib: error reading itkStimulateImageIO.cxx.o: file truncated

The identical clean build then passed, with no truncation and no short
object files. Evidence points at the container file-sharing layer rather
than this branch: 158 GB free at the time, the truncated archive was
exactly 49,152 bytes (12 × 4096, a dropped write tail rather than a
partial compile), and aarch64 is the only platform writing through a
Docker bind mount onto a macOS filesystem.

Docker Desktop updated itself between the two runs (29.7.2 → 29.8.0), so
the version cannot be credited with the fix, and one pass does not
disprove an intermittent fault. Recommended handling: retry a clean
build first; do not treat a truncation as a defect in this branch.

Commit series

Ten topical commits, no fixups and no merge commits. Bug fixes found
during platform testing were folded into the commits that introduced the
code, so the series reads as though the defects never existed.

Cavan Riley is credited via Co-Authored-By on the seven commits
carrying his original work.

[dzenanz]

I have taken a look. There is too much code to look carefully. I tried running locally:

M:\a\IPP>pixi run python3 "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
Error:   × could not find pixi.toml or pyproject.toml with tool.pixi at directory M:\a\IPP


M:\a\IPP>cd C:\Misc\ITKPythonPackage

M:\a\IPP>c:

C:\Misc\ITKPythonPackage>pixi run python3 "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
 WARN Encountered 2 warnings while parsing the manifest:
  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:82:1]
 81 │
 82 │ ╭─▶ [feature.manylinux228-build.system-requirements]
 83 │ │   linux = "5.4"
 84 │ ├─▶ libc = { family = "glibc", version = "2.28" }
    · ╰──── declare these on the `platforms` entries instead
 85 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:95:23]
 94 │ platforms = ["osx-arm64"]
 95 │ system-requirements = { macos = "14.0" }
    ·                       ─────────┬────────
    ·                                ╰── declare these on the `platforms` entries instead
 96 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

Error launching 'python3': The system cannot find the path specified. (os error 3)

C:\Misc\ITKPythonPackage>pixi run python "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
 WARN Encountered 2 warnings while parsing the manifest:
  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:82:1]
 81 │
 82 │ ╭─▶ [feature.manylinux228-build.system-requirements]
 83 │ │   linux = "5.4"
 84 │ ├─▶ libc = { family = "glibc", version = "2.28" }
    · ╰──── declare these on the `platforms` entries instead
 85 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:95:23]
 94 │ platforms = ["osx-arm64"]
 95 │ system-requirements = { macos = "14.0" }
    ·                       ─────────┬────────
    ·                                ╰── declare these on the `platforms` entries instead
 96 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

Running >>>>>: git fetch --tags origin  ; # in cwd=C:\Dev\ITK-git with check=False

Traceback (most recent call last):
  File "C:\Misc\ITKPythonPackage\scripts\build_wheels.py", line 628, in <module>
    build_wheels_main()
    ~~~~~~~~~~~~~~~~~^^
  File "C:\Misc\ITKPythonPackage\scripts\build_wheels.py", line 450, in build_wheels_main
    raise RuntimeError(
    ...<2 lines>...
    )
RuntimeError: Failed to fetch ITK tags in C:\Dev\ITK-git: fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.


C:\Misc\ITKPythonPackage>

[dzenanz]

And 3.10 is retained for ability to build ITK 5.4 wheels?

[dzenanz]

This seems to be doing something: C:\Misc\ITKPythonPackage>pixi run python scripts/build_wheels.py --platform-env windows-py311 --itk-git-tag v6.0b01

[dzenanz]

That failed too: run.log

[hjmjohnson]

Windows verification complete — handing back to Linux/macOS

Everything Windows-specific on this branch is now verified by real builds, and one cross-platform defect found along the way is fixed. Head is 1af2b15 (10 commits, no BUG: commits — every fix is folded into the commit that introduced the code). Coordination detail lives in the shared gist; this is the summary a reviewer needs.

What was verified on Windows 11 / VS 2022 (i7-9800X, 32 GB)

step	result
clean cache build from ef0c2d4 (pixi reinstalled, everything deleted first)	7 × cp311-abi3-win_amd64, cache 226,973,373 B, published to v6.0rc01.dev20260915, byte-identical
DLL bundling	9 DLLs in itk_core incl. itk_core.libs/tbb12.dll; wheels import in a clean venv with PATH stripped to System32 and ITK_EAGER_IMPORT=1
remote module (ITKBioCell) via the documented script, as a fresh consumer	step 06 reached; Requires-Dist: itk<7,>=6.0.0b1; repaired wheel (with msvcp140.dll) in the module's own dist/; pyproject.toml restored byte-identical
unit tests	97 passed on Windows (was 11 failed / 81 passed)

Defects fixed on this branch by the Windows work

All Windows-only unless noted; none were reachable from Linux/macOS testing.

1. TargetPlatform.detect() read PROCESSOR_ARCHITECTURE from the config mapping, not the environment — Unknown machine '' on any Windows build (POSIX hidden by os.uname).
2. Nested pixi run from inside an activated env overflowed the cmd.exe 8191-char limit (The input line is too long) — in the interpreter probe and in echo_check_call.
3. build_tarball() deleted the seven freshly built wheels from dist/ and reported rc=0.
4. pixi's vs2022 activation exports CMAKE_GENERATOR="Visual Studio 17 2022"; scikit-build-core honoured it, and the multi-config generator broke ITK wrapping's file(GENERATE) (castxml.inc "written multiple times"). Module wheels now pass -GNinja.
5. Cross-platform: the dependency rewrite emitted itk >= 6.0, < 7, which PEP 440 makes unsatisfiable by any 6.0.0bN (6.0.0b2 < 6.0); --pre cannot rescue it. Now >= 6.0.0b1, < 7.
6. delvewheel --add-path pointed at a non-existent oneTBB directory.
7. pixi global install aria2 always fails on win-64 and printed a red error that looked fatal.
8. A failed git checkout <tag> forced git checkout main, moving a cache-restored ITK source off the commit its binaries were built from.
9. 7-Zip -r treated build as a recursive pattern and swept a remote module's CMake tree into the cache; consumers then hit generator ... does not match.
10. Windows fixup_wheel ignored remote_module_wheel and wrote the repaired module wheel to the main dist/, leaving the unrepaired one as the module's only artifact.
11. Cross-platform (gist T7): on the --skip-itk-wheel-build path, 04_post_build_fixup and 05_final_import_test still ran against ITK wheels that were never built, and step 05 aborted the run before the module wheel. Both are now registered as _skipped no-ops when 03 is skipped; the step table was extracted into build_step_table() and is covered by tests. The earlier Windows "pass" of step 05 was an artifact of leftover wheels on the build machine — verified by reproducing the failure with dist/ empty, then re-running after the fix.

Plus three test fixes so the suite runs on Windows (os.uname monkeypatching, a cross-build test that inferred the host OS, a symlink that needs SeCreateSymbolicLinkPrivilege), and install-pixi.ps1 added to .gitignore.

For Linux/macOS

• T2 (Linux remote-module test) is unblocked by fix 11 — git fetch, rebuild from 1af2b15 or later. Expected Requires-Dist is itk<7,>=6.0.0b1, not the older >= 6.0, < 7.
• The published caches were built from ef0c2d4 and carry the pre-fix-11 scripts. The documented module scripts re-clone the branch when ITKPYTHONPACKAGE_TAG is set, so consumers get the fix; a consumer without that variable does not. Rebuilding caches is not urgent but is why cache and branch can disagree.
• Not on this branch: an experiment bounding ninja jobs by physical memory (-j = min(cpu, ram/4) with a link pool). Motivated by two OOM kills at -j16 on 32 GB; peak memory dropped to 9.9 GB, but the timing run was confounded and is parked on a separate local branch. Worth revisiting for CI runner sizing.