ENH: Python build system for ITK 6 wheels by CavRiley · Pull Request #302 · InsightSoftwareConsortium/ITKPythonPackage

CavRiley · 2026-03-29T01:46:27Z

Replaces the shell-script wheel build with a Python build system: pixi environments, a platform class hierarchy, and scripts that build ITK and remote-module wheels. Rebased onto main plus #308 and updated for ITK 6. Cavan Riley authored the original work and has moved on; his commits keep his authorship, and this branch carries it forward.

Depends on #308 (issue templates, split out of #302).

Review feedback on #302 and how it is addressed

@thewtex: drop manylinux2014 for ITK 6. Addressed. No manylinux2014 references remain outside pixi.lock.
@thewtex: are the Linux packages PyPI/manylinux compatible? manylinux228-py* environments produce PyPI-uploadable wheels; linux-py* environments produce linux_* wheels for local use. Utilities/scripts/publish_wheels.py now refuses to upload any linux_*-tagged wheel.
@dzenanz: removal of the quick start guide. The thread is resolved: this repository builds packages rather than teaching ITK Python usage. docs/index.rst now links the ITK Python quick start guide instead.
WIP: Macos python build script #127 (2019 "WIP: Macos python build script") is superseded by the macOS build path here.

Changes beyond #302

macOS arm64 wheels: delocate every wheel on all architectures, and export MACOSX_DEPLOYMENT_TARGET for the wheel build. Without this, arm64 wheels linked the build host's pixi libc++ and were tagged macosx_26_0. Verified: all wheels are macosx_11_0_arm64, bundle libc++, and import in a clean venv.
PERF: Set the default threader to Pool #305 port: ITK_DEFAULT_THREADER:STRING=Pool restored. ENH: Python build system for ITK 6 wheels #302 deleted every file PERF: Set the default threader to Pool #305 changed.
BUG: Avoid re-extracting the ITK build cache for module dependencies #307: dockcross-manylinux-build-module-deps.sh stays deleted. The Python dependency build reuses the shared ITK tree, so the EXDEV re-extraction bug does not apply.
Issue templates come from ENH: Add GitHub issue templates for bug reports and feature requests #308.
No CLAUDE.md: the original branch added one; it is left out of this PR.
Failed builds now fail: python -m build (ITK and remote-module wheels) and macOS delocate-wheel run with check=True. A failed remote-module build used to be recorded as a completed step, with exit code 0.
Python 3.11+ and Apple Silicon only: py310 environments and osx-64 removed, pixi.lock regenerated, requires-python >=3.11, and docs updated.
No stale default release: the download-and-build script requires ITK_PACKAGE_VERSION, because v6.0b02 caches use the old layout. ITK build and tarball scripts default to main.
Uploads: publish_wheels.py refuses linux_* wheels.
Remote module wheels declare the right ITK: the dependency rewrite now covers the bare itk metapackage (both == X.Y.* and ~= X.Y.Z forms), which five of the six wrapped modules use, and the floor is the numeric MAJOR.MINOR. Previously an ITK 6 module wheel was published with Requires-Dist: itk==5.4.*, and nothing reported it. The pin is bounded: itk >= 6.0, < 7, so a future ITK major cannot satisfy a wheel built against this ABI. Verified end to end by building ITKBioCell against the published cache.
Build caches are no longer lost: the cache is written to <build-dir-root>/dist, which container builds bind-mount. It previously went beside the build directory, so the aarch64 image reported a successful tarball step while the cache was discarded with the container. publish_tarball_cache.py looks in dist/ first and still accepts caches beside the build directory.

Follow-up work (not in this PR)

pixi-build, reusing the conda-forge libitk package for wrapping, and moving recipes into the ITK repository (raised by @thewtex and @CavRiley).

Validation

Built at ITK f1e39a5 on three platforms: manylinux_2_28 x86_64, manylinux_2_28 aarch64 (native Apple Silicon container), and macOS arm64. Each produced seven cp311-abi3 wheels and an ITKPythonBuilds tarball cache. macOS wheels were checked for bundled libc++, no build-host rpaths, and import in a clean venv. Windows is untested here. pre-commit passes on all files.

🤖 Generated with Claude Code

CavRiley · 2026-03-29T02:08:19Z

Also may close #127

dzenanz

I had a somewhat quick look.

thewtex · 2026-03-31T21:23:34Z

@CavRiley amazing effort!! 🥇 🔥

With a cursory review, I noted that we do not need to support manylinux2014 for ITK 6.

A few preliminary questions (I have not taken a detailed look):

Does it use pixi-build? Could we use pixi-build to build / re-use a C++ package? The motivation is to reduce build times and improve itk conda / conda-forge package compatibility.
Can we make it re-use the itk C++ conda-forge package?
Are the Linux packages compatible with pypi/manylinux?

See https://www.perplexity.ai/search/we-have-a-python-package-itk-t-MLAY4xnZRCeU9SGHhyJHQQ

CavRiley · 2026-04-01T15:15:37Z

Thank you, @thewtex!

With a cursory review, I noted that we do not need to support manylinux2014 for ITK 6.

Okay, I will remove this!

Does it use pixi-build? Could we use pixi-build to build / re-use a C++ package? The motivation is to reduce build times and improve itk conda / conda-forge package compatibility.

This does not use pixi-build, but I think it would be possible to use pixi-build to build the remote modules if not the base ITK modules as well. However, I would need to investigate this further to confirm it can handle ITK's CMake/SWIG wrapping.

Can we make it re-use the itk C++ conda-forge package?

Possibly, but I think the itk C++ conda-forge package would need to be built with the python wrapping (I'm not sure if this is currently the case) in order for it to be re-used here.

Are the Linux packages compatible with pypi/manylinux?

The linux-py* pixi environments produce wheels tagged linux_x86_64 which I don't think are uploadable to PyPI. For PyPI-compatible wheels, the manylinux228-py* environments with dockcross containers do make the wheels compatible with pypi/manylinux.

@thewtex

manylinux2014 is not needed for ITK 6. Remove the manylinux2014 image tag resolution, the manylinux1/manylinux2014 pixi-to-pattern renaming entries, and update shell script comments to reference _2_28 and _2_34 as the supported manylinux versions. Addresses review feedback from @thewtex on PR #302. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@thewtex

manylinux2014 is not needed for ITK 6. Remove the manylinux2014 image tag resolution, the manylinux1/manylinux2014 pixi-to-pattern renaming entries, and update shell script comments to reference _2_28 and _2_34 as the supported manylinux versions. Addresses review feedback from @thewtex on PR InsightSoftwareConsortium#302. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

CavRiley · 2026-04-12T21:33:15Z

Hey @thewtex, just following up on this.

Is https://github.com/conda-forge/itk-feedstock the home for all ITK conda packages now? It looks like the itk Python conda packages currently repackage the PyPI wheels rather than build from source, would it be worth building them from source instead?

We've been working on a libitk-wrapping conda package that @hjmjohnson created to bundle ITK C++ libraries with full Python wrapping artifacts. With this as a build dependency, remote modules could use Pixi build to generate their conda Python packaging (possibly with a new GitHub action from ITKRemoteModuleBuildTestPackageAction).

I think it makes sense to have the recipe infrastructure for this (and potentially the other ITK conda packages) in the main ITK repository, so the recipes stay in sync with ITK releases. From there it could either be submitted to conda-forge as a new feedstock, or added as an output to the existing libitk-feedstock.

thewtex · 2026-04-14T19:35:27Z

@CavRiley thanks for taking a look.

Sorry, I meant the libitk-feedstock instead of the itk-feedstock.

Yes, I think we are all aligning on the same goals -- we want the python wrapping to use a libitk-equivalent C++ pixi package / conda-forge package, locally or in remote module CI or on conda-forge. And yes, it would be ideal to move this to the ITK repository. 💯

The name conda-packages/ more clearly describes the contents (a multi-output recipe producing libitk, libitk-devel, and libitk-wrapping) and avoids confusion with the single-recipe convention of standalone conda-forge feedstocks. Update all self-referential paths in recipe.yaml. Add conda-packages/README.md covering the three-package split and install components, rattler-build vs conda-build format differences, CMake flag alignment and divergence from the conda-forge libitk feedstock, the bug fixes required for system library builds and their root causes, platform matrix, the build guard for fast iteration, and the relationship to InsightSoftwareConsortium/ITKPythonPackage#302. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Give reporters a structured form for build and packaging problems so issues arrive with the platform, Python version, ITK version, and entry point needed to reproduce them. Point general questions to ITK Discourse and the ITKPythonPackage documentation. Split out of the Python build system work in InsightSoftwareConsortium#302, which it does not depend on; issue titles use the ITK commit prefixes this repository enforces. Co-Authored-By: Cavan Riley <127157889+CavRiley@users.noreply.github.com>

…n scripts Replaces the primary build orchestration layer with a Python-based system. Adds core abstractions including BuildManager for step-wise build persistence/resumption, CMakeArgumentBuilder for managing CMake definitions, cross-platform venv utilities, subprocess helpers, and pixi environment integration. Establishes platform-specific base classes that Linux, macOS, and Windows implementations extend. Restructures CMake layout into modular BuildWheelsSupport and SuperbuildSupport components. Shell scripts remain where still required. Co-authored-by: Hans J. Johnson <hans-johnson@uiowa.edu>

A wheel compiled against ITK 6.0 must require >= 6.0 at install time, not >= 5.4. The floor reflects the ITK series the wheel was linked against. The build system supports building against any ITK from v5.4 through the latest — the floor is derived dynamically from ITK_PACKAGE_VERSION at build time.

…main Add shell and Python scripts that clone ITK, ITKPythonPackage, and all remote modules from their latest main branches, then build ITK Python wheels followed by every remote module that has Python wrapping. All wheels are collected into a single /tmp/<timestamp>_LatestITKPython/dist directory. Both scripts parse ITK's Modules/Remote/*.remote.cmake files to discover remote module git repositories, filter to those with wrapping/ and pyproject.toml, and build each module wheel reusing the ITK build from the first step. Usage: ./scripts/build-all-latest-wheels.sh [platform-env] python scripts/build_all_latest_wheels.py --platform-env linux-py311

Add --itk-ref option to both shell and Python build-all-latest-wheels scripts. Defaults to "main" but accepts any git ref (branch name, tag like v6.0b02, or commit hash). The ITK clone is now a full clone (not shallow) to support arbitrary refs.

Run the full pre-commit suite repo-wide (black, ruff, shfmt, taplo) and resolve shellcheck findings: egrep -> grep -E, guard rm with ${MODULES_DIR:?}, and count wheels with find instead of ls.

…target macOS arm64 wheels built by the pixi toolchain were unusable outside the build host and carried the wrong platform tag. Every extension links @rpath/libc++.1.dylib from the conda-forge toolchain, with the pixi environment as the only LC_RPATH. delocate ran only for x86_64, and the base-class fixup_wheels() only visits itk_core, so no wheel bundled libc++ and `import itk` failed on any other Mac. The final import test passed only because it ran on the build host. The wheel step also ran without MACOSX_DEPLOYMENT_TARGET in its environment, so scikit-build-core tagged the wheels with the host macOS version (macosx_26_0) even though CMake and every binary target 11.0. Export MACOSX_DEPLOYMENT_TARGET for the wheel build, and run delocate-wheel --require-archs <arch> on every wheel in dist/ for both architectures. Verified on an ITK f1e39a5 arm64 build: all wheels are tagged macosx_11_0_arm64, bundle itk/.dylibs/libc++.1.0.dylib, and import in a clean Python 3.12 venv without pixi.

PR InsightSoftwareConsortium#305 set ITK_DEFAULT_THREADER=Pool for ITK Python wheels, but only in the shell scripts and top-level CMakeLists.txt that the Python build system replaces. None of the replacement files carried the setting, so wheels built by it silently fell back to ITK's default threader. Set ITK_DEFAULT_THREADER:STRING=Pool in the ITK wrapping configuration used for direct ITK builds and in the SuperBuild ITK ExternalProject arguments.

The linux-py* pixi environments build native wheels tagged linux_x86_64 or linux_aarch64, which PyPI rejects. publish_wheels.py uploaded every wheel in the dist directory, so a mixed directory would fail partway through an upload after its manylinux wheels were already published. Check platform tags before twine runs and stop with a message naming the offending wheels and the manylinux228-py* environments to use.

The quick start guide was removed from this repository because the ITK documentation already covers it; link to that guide so readers looking for ITK Python usage still find it from here.

echo_check_call does not raise on a non-zero exit by default, so a failed `python -m build` for the ITK wheels or a remote module, or a failed delocate-wheel, was recorded as a completed step and the driver exited 0. A remote-module build that errored therefore looked green. Pass check=True for those commands so the failure raises, BuildManager marks the step failed, and the build stops.

ITK 6 wheels use the CPython 3.11 Stable ABI, and Intel macOS is no longer supported. Remove the py310 pixi environments and the osx-64 platform, regenerate pixi.lock, raise requires-python and the tool target versions to 3.11, default the build scripts to py311, and update the docs, help text, and pull request template to match. Drop the now unreachable Python < 3.11 branch in pyproject_configure.py. The scripts defaulted ITK_PACKAGE_VERSION to v6.0b02, whose ITKPythonBuilds caches predate this build layout and cannot be used. The download-and-build script now requires ITK_PACKAGE_VERSION, and the ITK build and tarball scripts default the ITK source to main.

delocate-wheel --require-archs arm64 raises DelocationError on a wheel that contains no binaries, such as the pure-Python itk metapackage, so the fixup step failed once its failures stopped being ignored. Check the wheel for .so/.dylib members and skip delocate when it holds none.

The aarch64 path runs the manylinux image directly and mounted only /work/ITKPythonPackage and /work/ITKPythonPackage-build. The tarball step writes the build cache to /work itself, so the cache was created inside the container and discarded with --rm; aarch64 caches could never be produced. The x64 path does not have this problem because the dockcross runner mounts the host working directory as /work. Mount the directory holding the checkout and the build tree as /work, matching dockcross.

This reverts commit 7ebe976. Mounting the parent directory at /work while /work/ITKPythonPackage-build and /work/ITKPythonPackage are bind-mounted beneath it resolves non-deterministically on Docker Desktop: one run wrote its build log and CMake configure tree into a sibling build directory that happened to be named ITKPythonPackage-build on the host, instead of the mounted target. Fix the lost cache by writing it inside the mounted build tree instead.

create_posix_tarball wrote ITKPythonBuilds-*.tar.zst next to the build directory. Container builds bind-mount the build tree but not its parent, so the aarch64 image reported a successful tarball step while the cache was discarded with the container; aarch64 caches could never be produced. Write the cache to <build-dir-root>/dist, which every platform mounts and which the archive already excludes, so it cannot capture itself. The archive root and member paths are unchanged. publish_tarball_cache.py looks in dist/ first and still accepts caches beside the build directory.

create_posix_tarball left the intermediate ITKPythonBuilds-*.tar beside the compressed cache, several GB per build that nothing consumes; only the .tar.zst is published. Delete it once zstd has succeeded, so a failed compression still leaves the archive in place.

"-xr!Wrapping\Modules" contains \M, which is not a recognised escape sequence: Python emits a SyntaxWarning today and will raise an error in a future release. The literal's runtime value is unchanged. Mark that literal and the surrounding docstring as raw, and select ruff's W rules so W605 catches invalid escape sequences from now on.

Strategy 1 rewrote only the six ITK sub-package pins, but most remote modules depend on the itk metapackage instead: five of the six wrapped modules that declare dependencies pin "itk == 5.4.*" or "itk~=5.4.0". Their ITK 6 wheels were published with Requires-Dist: itk==5.4.*, which pip cannot satisfy against the ITK 6 wheels they were compiled with, and nothing reported it because no pattern matched. Rewrite the metapackage as well, accepting both the == X.Y.* and ~= X.Y.Z pin forms. Remote module cross-dependencies such as itk-minimalpathextraction are still left alone.

The floor came from splitting the version on ".", so a tag such as v6.0rc01.dev20260915 produced "itk >= v6.0rc01" and v5.4.7 produced "itk >= v5.4". Pinning a remote module wheel to one release candidate also invites pip to consider ITK pre-releases for that dependency, when the wheel works with the whole 6.0 series. Take the leading MAJOR.MINOR digits instead, so the same tag yields "itk >= 6.0"; a value that is not a version, such as a branch name, is still passed through unchanged.

The rewritten pin had no upper bound, so a wheel compiled against ITK 6 was satisfiable by ITK 7 or 8 with an incompatible ABI. Emit "itk >= 6.0, < 7" instead, and leave a module's pins alone when the ITK version is not MAJOR.MINOR, since no valid specifier can be built from a branch name. post_build_cleanup interpolated the whole package_env_config dict into two globs and searched the source checkout, so it matched nothing and silently removed neither the ITK build tree nor the caches. Point them at <build-dir-root>/build and <build-dir-root>/dist. Correct two docstrings that described the behaviour before it changed: delocate now repairs every wheel on both architectures, and a module dependency builds against the ITK tree already in use rather than re-extracting the cache.

Both cache scripts tell a developer running them from the wrong directory to "git checkout python_based_build_scripts", a branch that does not carry this build system; the work lives on python-build-system. Anyone following that message lands on the wrong code.

CONTRIBUTING.md told contributors to write Conventional Commits, but commitizen enforces the ITK prefixes that every commit here uses, so the documented rule rejected the commits it asked for. The remote-module cross-dependency warning only recognised "== X.Y.*", so a cross-dep pinned as "~= X.Y.Z" was neither rewritten nor reported. Delete the commented-out functions in wheel_builder_utils.py (push_env, debug, parse_kv_overrides, get_git_id, cmake_compiler_defaults), none of which has a caller, and hoist the function-local imports to module scope, dropping the tomli fallback that a Python 3.11+ project can never reach.

build-all-latest-wheels.sh and build_all_latest_wheels.py drive the same build with the same five options, on a branch whose purpose is to replace shell scripts with Python. Nothing references the shell copy. Keep the Python driver, whose defaults already point at this branch.

hjmjohnson · 2026-09-17T23:23:05Z

Superseded by #309.

#309 carries the same work, re-cut from main as twelve reviewable commits,
plus fixes for sixteen defects that surfaced while building it on Linux
x86_64, Linux aarch64, and macOS arm64. The ones worth knowing about if you
reviewed this branch:

A failed auditwheel repair deleted the original wheel anyway, so a repair
failure destroyed the artifact rather than reporting it. On x86_64 this left
a single 12 KB meta-wheel and a build that reported success.
05_final_import_test could not fail: it printed "Documentation tests
passed." unconditionally while discarding every check's exit status.
The macOS itk meta-wheel was tagged macosx_10_7_arm64, a tag pip rejects
on every arm64 Mac — so the package users actually install was
uninstallable.
A failed ITK cmake/ninja was recorded done and, because resumed builds
skip completed steps, was never retried.

Two decisions also changed and are flagged at the top of #309: Linux wheels
now carry exactly one platform tag (auditwheel --only-plat), and the macOS
deployment target is 14.0 rather than 11.0.

All commits retain their original Co-Authored-By attribution.

Windows remains unbuilt in both branches.

Leaving this open rather than closing it, in case you want the two side by
side during review.

hjmjohnson · 2026-09-20T13:58:04Z

Closing in favor of #309, which carries this work forward. Thank you
@CavRiley — the design here is the basis of what landed, and the 41
commits on this branch are what made the re-cut possible.

#309 is the same build system, re-cut from main as 11 reviewable
commits with fixes folded into the commits that introduced the code. It
is still WIP: while refinements continue, but it now builds on all
four target platforms.

Status of the successor

Platform	Clean build	Remote-module end-to-end
Linux x86_64	pass	pass
Linux aarch64	pass	—
macOS arm64	pass	—
Windows x86_64	pass	pass

All four build caches on v6.0rc01.dev20260915 have been rebuilt and
published from the current code. 106 unit tests pass and
pre-commit run --all-files exits 0.

Carried over from the review discussion on this PR: manylinux2014 is
dropped, Python 3.11 is the floor, and wheels are stable-ABI
(cp311-abi3) only — one wheel per platform.

CavRiley force-pushed the python-build-system branch from 182e8a5 to 4aa0abb Compare March 29, 2026 02:05

CavRiley marked this pull request as ready for review March 29, 2026 22:06

CavRiley force-pushed the python-build-system branch from 1ea2b13 to db39910 Compare March 30, 2026 01:02

dzenanz reviewed Mar 30, 2026

View reviewed changes

Comment thread docs/Quick_start_guide.rst

CavRiley force-pushed the python-build-system branch 3 times, most recently from 4d78860 to 4288703 Compare March 31, 2026 16:15

CavRiley force-pushed the python-build-system branch from 4288703 to 86b1e2f Compare April 1, 2026 21:10

hjmjohnson marked this pull request as draft April 3, 2026 15:35

hjmjohnson force-pushed the python-build-system branch 3 times, most recently from 03cc291 to 221359d Compare April 3, 2026 20:54

hjmjohnson mentioned this pull request Apr 17, 2026

WIP: CI PLATFORM TESTING CONDAFORGE...DELETEME InsightSoftwareConsortium/ITK#6081

Closed

This was referenced May 20, 2026

Add libitk wrapping output conda-forge/libitk-feedstock#118

Closed

Add from source wrapping conda-forge/itk-feedstock#50

Merged

CavRiley force-pushed the python-build-system branch 3 times, most recently from 61a3bb4 to 54cbec6 Compare June 25, 2026 16:01

N-Dekker mentioned this pull request Jul 2, 2026

PERF: Set the default threader to Pool #305

Merged

hjmjohnson mentioned this pull request Sep 15, 2026

ENH: Add GitHub issue templates for bug reports and feature requests #308

Merged

hjmjohnson and others added 16 commits September 16, 2026 06:45

STYLE: Move build-all-latest-wheels scripts to Utilities/scripts

a5c5234

STYLE: Move publish scripts to Utilities/scripts

48eac36

STYLE: Apply pre-commit hook fixes across the repository

d1b872b

Run the full pre-commit suite repo-wide (black, ruff, shfmt, taplo) and resolve shellcheck findings: egrep -> grep -E, guard rm with ${MODULES_DIR:?}, and count wheels with find instead of ls.

DOC: Link the ITK Python quick start guide from the docs landing page

b4d6553

The quick start guide was removed from this repository because the ITK documentation already covers it; link to that guide so readers looking for ITK Python usage still find it from here.

hjmjohnson force-pushed the python-build-system branch from e124c99 to 9099c9c Compare September 16, 2026 11:46

hjmjohnson changed the title ~~Python Build System~~ ENH: Python build system for ITK 6 wheels Sep 16, 2026

hjmjohnson added 8 commits September 16, 2026 06:53

hjmjohnson mentioned this pull request Sep 17, 2026

WIP: ENH: Replace the shell wheel build system with Python (supersedes #302) #309

Draft

hjmjohnson closed this Sep 20, 2026

Navigation Menu

ENH: Python build system for ITK 6 wheels - #302

ENH: Python build system for ITK 6 wheels#302
CavRiley wants to merge 41 commits into
InsightSoftwareConsortium:mainfrom
BRAINSia:python-build-system

CavRiley commented Mar 29, 2026 •

edited by hjmjohnson

Loading

Uh oh!

CavRiley commented Mar 29, 2026

Uh oh!

dzenanz left a comment

Uh oh!

Uh oh!

thewtex commented Mar 31, 2026

Uh oh!

CavRiley commented Apr 1, 2026

Uh oh!

CavRiley commented Apr 12, 2026

Uh oh!

thewtex commented Apr 14, 2026

Uh oh!

hjmjohnson commented Sep 17, 2026

Uh oh!

hjmjohnson commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

CavRiley commented Mar 29, 2026 • edited by hjmjohnson Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CavRiley commented Mar 29, 2026

Uh oh!

dzenanz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thewtex commented Mar 31, 2026

Uh oh!

CavRiley commented Apr 1, 2026

Uh oh!

CavRiley commented Apr 12, 2026

Uh oh!

thewtex commented Apr 14, 2026

Uh oh!

hjmjohnson commented Sep 17, 2026

Uh oh!

hjmjohnson commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CavRiley commented Mar 29, 2026 •

edited by hjmjohnson

Loading