Repository navigation
fix(pypi): fetch retained wheel archives lazily - #4316
Open
charleysaintNV wants to merge 3 commits into
Open
charleysaintNV wants to merge 3 commits into
charleysaintNV wants to merge 3 commits into
Conversation
aignas
reviewed
Oct 23, 2026
aignas
left a comment
Collaborator
There was a problem hiding this comment.
Thank you for the PR.
So all things added up, this way is nice, because:
- it retains the whl downloaded via bazel_downloader in the repo cache.
- it extracts and removes the whl and if the user is not using
whltargets then the user may be paying less cost for storing the extracted wheels in the reproducible repo cache. - If the user asks for the whl target, we should take it from the bazel_downloader cache without a problem.
- This may facilitate migrating to whl extraction in the build phase.
Things that are not so good or should be discussed:
- Should we do patching in the extra
_whlrepository? Then we extract the source from the extra_whlrepository and do not do any patching in the main repo that haspkg? - Naming: should we name the repo
whl_file_repoinstead ofwhl_download? - Gazelle is pulling in wheels to get top level imports. I think we should instead change it to use
py_library. That would work better with the promises of this PR and users would not need to pay the cost of extracting the wheels there. The extraction is cached in the build cache, but I wonder how this should be done.
| @@ -0,0 +1,4 @@ | |||
| Downloaded wheels are removed after extraction when their original archive can | |||
Collaborator
There was a problem hiding this comment.
Suggested change
| Downloaded wheels are removed after extraction when their original archive can | |
| (pypi) Downloaded wheels are removed after extraction when their original archive can |
Comment on lines
47
to
+48
| common --deleted_packages=tests/integration/whl_library | ||
| common --deleted_packages=tests/integration/whl_storage |
Collaborator
There was a problem hiding this comment.
We could combine these 2 together. Having an extra test has a non-zero cost.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
URL-resolved wheels currently retain both the downloaded archive and extracted files even when a build only consumes the Python library. Download the archive temporarily for extraction and remove it afterward, while preserving the existing
whlandwhl_filetargets through a lazy sibling<repo>_whlrepository. Ordinary Python builds do not fetch that sibling repository.Keep patched replacement wheels local and preserve caller-owned wheel inputs. Source-built wheels and direct
whl_archiveURL calls retain their current archive behavior. Both download paths share checksum validation, authentication, and URL/environment substitution.Add an integration test that generates a wheel locally, serves it over HTTP, and runs separate Bazel invocations to check archive cleanup, lazy repository creation, both wheel targets, matching and nonmatching patches, and preservation of caller-owned inputs. An additional case exercises the full
pip.parse(uv_lock = ...)path, verifying import, absence of the archive repository, and subsequent byte-identical archive retrieval through:whl. Add analysis coverage for archive ownership and exclusion from runtime files.Validation:
--config=fast-tests.//tests/integration:whl_storage_test_bazel_selfpasses all four test methods. This single nested test is run with the fast-tests tag filters overridden and local test concurrency limited to one.whltarget materializes the archive. Its extracted repository uses 57.74 MiB instead of 73.61 MiB allocated on the test filesystem. Shared download caches are outside that measurement.Implements the delete-by-default / optional-archive-access suggestion in #3091 (comment). Preserves the wheel-consumer functionality introduced in #364 and fixed in #458. Related to #3968 and complements the extraction-reuse work in #3856.