Fix ensurepip bundled pip upgrade for Python 3.12+#38765
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a failure in the Docker image build process for Python 3.12+ environments. By isolating the ensurepip upgrade logic based on the Python version, the changes ensure that the bundled pip wheel is correctly updated without triggering errors caused by the absence of setuptools in newer Python releases. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new Python script, upgrade_bundled_pip.py, and updates the Dockerfile to handle upgrading bundled pip wheels in ensurepip for Python 3.12+ (which no longer bundles setuptools), while maintaining the existing upgrade_ensurepip flow for Python 3.10 and 3.11. The reviewer suggests unifying the upgrade logic for all Python versions (3.10+) by extending the custom Python script to handle setuptools and clean up old wheels. This improvement would eliminate the external upgrade_ensurepip dependency and significantly simplify the Dockerfile's shell commands.
| if [ "${py_version}" = "3.10" ] || [ "${py_version}" = "3.11" ]; then \ | ||
| find /usr/local/lib/python${py_version}/ensurepip/_bundled/setuptools-* -type f ! -name $(basename $(ls -v /usr/local/lib/python${py_version}/ensurepip/_bundled/setuptools-*-py3-none-any.whl | tail -n 1)) -delete; \ | ||
| fi; \ | ||
| find /usr/local/lib/python${py_version}/ensurepip/_bundled/pip-* -type f ! -name $(basename $(ls -v /usr/local/lib/python${py_version}/ensurepip/_bundled/pip-*-py3-none-any.whl | tail -n 1)) -delete; \ | ||
| pip uninstall upgrade_ensurepip -y; \ | ||
| python3 -m ensurepip; | ||
| pip install upgrade_ensurepip && \ | ||
| python3 -m upgrade_ensurepip && \ | ||
| find /usr/local/lib/python${py_version}/ensurepip/_bundled/setuptools-* -type f ! -name $(basename $(ls -v /usr/local/lib/python${py_version}/ensurepip/_bundled/setuptools-*-py3-none-any.whl | tail -n 1)) -delete && \ | ||
| find /usr/local/lib/python${py_version}/ensurepip/_bundled/pip-* -type f ! -name $(basename $(ls -v /usr/local/lib/python${py_version}/ensurepip/_bundled/pip-*-py3-none-any.whl | tail -n 1)) -delete && \ | ||
| pip uninstall upgrade_ensurepip -y && \ | ||
| python3 -m ensurepip; \ | ||
| else \ | ||
| python3 /tmp/upgrade_bundled_pip.py && \ | ||
| find /usr/local/lib/python${py_version}/ensurepip/_bundled/pip-* -type f ! -name $(basename $(ls -v /usr/local/lib/python${py_version}/ensurepip/_bundled/pip-*-py3-none-any.whl | tail -n 1)) -delete && \ | ||
| python3 -m ensurepip; \ | ||
| fi |
There was a problem hiding this comment.
Instead of maintaining separate logic for Python 3.10/3.11 (using the external upgrade_ensurepip package) and Python 3.12+ (using a custom script), we can unify both cases by extending upgrade_bundled_pip.py to handle setuptools when it is present. This completely eliminates the external upgrade_ensurepip dependency, avoids complex shell-based find/ls -v cleanup commands, and simplifies the Dockerfile significantly.
python3 /tmp/upgrade_bundled_pip.py && \
python3 -m ensurepip;
| def main(): | ||
| ep_path = Path(ensurepip.__file__) | ||
| wheel_dir = ep_path.parent / '_bundled' | ||
| pip_version = subprocess.check_output( | ||
| [sys.executable, '-m', 'pip', '--version'], | ||
| text=True).split()[1] | ||
| subprocess.check_call([ | ||
| sys.executable, | ||
| '-m', | ||
| 'pip', | ||
| 'download', | ||
| 'pip=={}'.format(pip_version), | ||
| '-d', | ||
| str(wheel_dir), | ||
| '--no-deps', | ||
| ]) | ||
| lines = ep_path.read_text().splitlines() | ||
| pip_line = None | ||
| for idx, line in enumerate(lines): | ||
| if line.startswith('_PIP_VERSION = '): | ||
| pip_line = idx | ||
| break | ||
| if pip_line is None: | ||
| sys.exit('ensurepip _PIP_VERSION not found') | ||
| org = ep_path.with_suffix('.py.org') | ||
| if not org.exists(): | ||
| ep_path.rename(org) | ||
| lines[pip_line] = '_PIP_VERSION = "{}"'.format(pip_version) | ||
| ep_path.write_text('\n'.join(lines) + '\n') | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
| main() |
There was a problem hiding this comment.
We can enhance upgrade_bundled_pip.py to support both pip and setuptools (if bundled), and handle the cleanup of old wheels directly in Python. This makes the script fully self-contained, robust, and reusable across all Python versions (3.10+), eliminating the need for complex shell logic in the Dockerfile.
def main():
ep_path = Path(ensurepip.__file__)
wheel_dir = ep_path.parent / '_bundled'
# Get current pip version
pip_version = subprocess.check_output(
[sys.executable, '-m', 'pip', '--version'],
text=True).split()[1]
# Download new pip wheel
subprocess.check_call([
sys.executable,
'-m',
'pip',
'download',
f'pip=={pip_version}',
'-d',
str(wheel_dir),
'--no-deps',
])
org = ep_path.with_suffix('.py.org')
if org.exists():
lines = org.read_text().splitlines()
else:
lines = ep_path.read_text().splitlines()
ep_path.rename(org)
# Update pip version in ensurepip/__init__.py
pip_line = None
for idx, line in enumerate(lines):
if line.startswith('_PIP_VERSION = '):
pip_line = idx
break
if pip_line is None:
sys.exit('ensurepip _PIP_VERSION not found')
lines[pip_line] = f'_PIP_VERSION = "{pip_version}"'
# Check if setuptools is also bundled (Python < 3.12)
setuptools_line = None
setuptools_version = None
for idx, line in enumerate(lines):
if line.startswith('_SETUPTOOLS_VERSION = '):
setuptools_line = idx
break
if setuptools_line is not None:
try:
setuptools_version = subprocess.check_output(
[sys.executable, '-c', 'import setuptools; print(setuptools.__version__)'],
text=True).strip()
subprocess.check_call([
sys.executable,
'-m',
'pip',
'download',
f'setuptools=={setuptools_version}',
'-d',
str(wheel_dir),
'--no-deps',
])
lines[setuptools_line] = f'_SETUPTOOLS_VERSION = "{setuptools_version}"'
except Exception as e:
print(f"Failed to upgrade bundled setuptools: {e}", file=sys.stderr)
# Write modified ensurepip/__init__.py
ep_path.write_text('\n'.join(lines) + '\n')
# Clean up old wheels in _bundled
for path in wheel_dir.glob('pip-*.whl'):
if pip_version not in path.name:
path.unlink()
if setuptools_version is not None:
for path in wheel_dir.glob('setuptools-*.whl'):
if setuptools_version not in path.name:
path.unlink()
if __name__ == '__main__':
main()
|
Assigning reviewers: R: @claudevdm for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
1 participant
Fixes: #38763
Republish Released Docker Images workflow is failing on the pushAll job (Python 3.14). python3 -m upgrade_ensurepip crashes because Python 3.12+ no longer bundles setuptools in ensurepip but the Dockerfile still run that tool for all versions
So the fix is keeping upgrade_ensurepip for 3.10/3.11 only and for 3.12+, will run a small license_scripts/upgrade_bundled_pip.py helper to update only the bundled pip wheel, then run the existing pip cleanup and ensurepip.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.