COMP: Bootstrap pip via ensurepip on Windows wheel build to fix Python 3.10+#304
Open
hjmjohnson wants to merge 1 commit intomainfrom
Open
COMP: Bootstrap pip via ensurepip on Windows wheel build to fix Python 3.10+#304hjmjohnson wants to merge 1 commit intomainfrom
hjmjohnson wants to merge 1 commit intomainfrom
Conversation
…n 3.10+
scikit-ci-addons' install-python.ps1 downloads get-pip.py from a pinned
GitHub gist (jcfr commit 8478d43e) rather than the canonical
bootstrap.pypa.io URL, and installs a pip old enough to vendor an
html5lib that still uses `from collections import Mapping`. That
import was removed in Python 3.10, so the very first invocation of
`python -m pip install pip --upgrade` (the bootstrap step in
`build_wheels`) dies with:
File ".../pip.zip/pip/_vendor/html5lib/_trie/_base.py", line 3
ImportError: cannot import name 'Mapping' from 'collections'
The failure is consistent on Python 3.10 and 3.11 across all Windows
matrix jobs (build-windows-python-packages (10) and (11)).
Fix it by running `python -m ensurepip --upgrade --default-pip` BEFORE
the existing `pip install pip --upgrade`. ensurepip uses the pip wheel
that CPython itself bundled at release time (>= 21.x for Python 3.10),
which has a working html5lib >= 1.1, and the subsequent self-upgrade
then proceeds normally.
The PowerShell entry point (windows-download-cache-and-build-module-wheels.ps1)
still IEXes the upstream install-python.ps1 unchanged - the chicken-and-egg
fix is purely Python-side, so no interaction with scikit-ci-addons is
required. Companion `windows_build_wheels.py` is unchanged because its
pip operations go through a virtualenv created by `virtualenv.exe`, and
modern virtualenv ships its own bundled pip wheels rather than copying
the (broken) system pip.
Surfaced by KitwareMedical/ITKUltrasound#246, which is the first remote
module PR to actually exercise the python wheel build matrix on Python
3.10+ after a related CI fix made the workflow start running again.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merged
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bootstrap pip via
python -m ensurepip --upgrade --default-pipbefore invoking the system pip inwindows_build_module_wheels.py. Sidesteps a Python 3.10+ chicken-and-egg failure where the system pip's vendoredhtml5libdoesfrom collections import Mapping(removed in 3.10) and dies before anypip installcan run.Surfaced by KitwareMedical/ITKUltrasound#246.
Failure trace
Consistent on Python 3.10 and 3.11 across all
build-windows-python-packagesmatrix jobs.Root cause
scripts/windows-download-cache-and-build-module-wheels.ps1IEXes the upstreaminstall-python.ps1fromscikit-build/scikit-ci-addons. That script's line 160 downloadsget-pip.pyfrom a pinned GitHub gist (jcfr/db7347e8708b9f32d45ab36125fad6d3@8478d43e) rather thanhttps://bootstrap.pypa.io/get-pip.py. The pinned gist installs a pip old enough to vendor an html5lib that still usesfrom collections import Mapping.When
windows_build_module_wheels.py:55runspython -m pip install pip --upgrade, the system pip loads its_vendor/html5lib._trie._baseand dies before any package-resolution logic runs. The bug cannot be worked around using the broken pip itself.Why ensurepip works
ensurepipinstalls the pip wheel that CPython itself bundled at release time (Lib/ensurepip/_bundled/), which is ≥ 21.x for Python 3.10 and ≥ 23.x for 3.11 — both ship a workinghtml5lib≥ 1.1 with thecollections.abc.Mappingmigration applied. Once ensurepip writes a fresh pip into the system Python, the subsequentpip install pip --upgradefinds a working pip and proceeds normally.What is NOT changed
windows-download-cache-and-build-module-wheels.ps1: still IEXes the upstreaminstall-python.ps1unchanged. The fix is purely Python-side; no scikit-ci-addons interaction needed.scripts/windows_build_wheels.py: unchanged. Its pip operations go through a virtualenv created byvirtualenv.exe, and modern virtualenv ≥ 20.x ships its own bundled pip seed wheels instead of copying the (broken) system pip into the new venv.scikit-ci-addonspinned gist URL: ideally fixed at the source eventually (replace withbootstrap.pypa.io/get-pip.py), but that's a separate, broader-scope change in a different repo.Test plan
build-windows-python-packages (10)and(11)jobs progress pastpip installinto actual C++ compilation