Skip to content

fix(pypi): fetch retained wheel archives lazily - #4316

Open
charleysaintNV wants to merge 3 commits into
bazel-contrib:mainfrom
charleysaintNV:fix/lazy-wheel-archives
Open

charleysaintNV wants to merge 3 commits into
bazel-contrib:mainfrom
charleysaintNV:fix/lazy-wheel-archives

Conversation

@charleysaintNV

@charleysaintNV charleysaintNV commented Oct 22, 2026 •

Copy link
Copy Markdown

URL-resolved wheels currently retain both the downloaded archive and extracted files even when a build only consumes the Python library. Download the archive temporarily for extraction and remove it afterward, while preserving the existing whl and whl_file targets through a lazy sibling <repo>_whl repository. Ordinary Python builds do not fetch that sibling repository.

Keep patched replacement wheels local and preserve caller-owned wheel inputs. Source-built wheels and direct whl_archive URL calls retain their current archive behavior. Both download paths share checksum validation, authentication, and URL/environment substitution.

Add an integration test that generates a wheel locally, serves it over HTTP, and runs separate Bazel invocations to check archive cleanup, lazy repository creation, both wheel targets, matching and nonmatching patches, and preservation of caller-owned inputs. An additional case exercises the full pip.parse(uv_lock = ...) path, verifying import, absence of the archive repository, and subsequent byte-identical archive retrieval through :whl. Add analysis coverage for archive ownership and exclusion from runtime files.

Validation:

  • 23 focused Bazel regression targets pass with --config=fast-tests.
  • //tests/integration:whl_storage_test_bazel_self passes all four test methods. This single nested test is run with the fast-tests tag filters overridden and local test concurrency limited to one.
  • The storage analysis suite passes under bzlmod. Under Bazel 7.4.1 WORKSPACE mode, the bzlmod-specific ownership assertion is skipped and runtime archive exclusion passes.
  • All applicable pre-commit hooks pass, including buildifier, Ruff, deleted-package generation, and Gazelle.
  • The NumPy 2.3.3 PoC imports successfully without materializing an archive repository; explicitly building its whl target materializes the archive. Its extracted repository uses 57.74 MiB instead of 73.61 MiB allocated on the test filesystem. Shared download caches are outside that measurement.

Implements the delete-by-default / optional-archive-access suggestion in #3091 (comment). Preserves the wheel-consumer functionality introduced in #364 and fixed in #458. Related to #3968 and complements the extraction-reuse work in #3856.

@aignas aignas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR.

So all things added up, this way is nice, because:

  • it retains the whl downloaded via bazel_downloader in the repo cache.
  • it extracts and removes the whl and if the user is not using whl targets then the user may be paying less cost for storing the extracted wheels in the reproducible repo cache.
  • If the user asks for the whl target, we should take it from the bazel_downloader cache without a problem.
  • This may facilitate migrating to whl extraction in the build phase.

Things that are not so good or should be discussed:

  • Should we do patching in the extra _whl repository? Then we extract the source from the extra _whl repository and do not do any patching in the main repo that has pkg?
  • Naming: should we name the repo whl_file_repo instead of whl_download?
  • Gazelle is pulling in wheels to get top level imports. I think we should instead change it to use py_library. That would work better with the promises of this PR and users would not need to pay the cost of extracting the wheels there. The extraction is cached in the build cache, but I wonder how this should be done.

@@ -0,0 +1,4 @@
Downloaded wheels are removed after extraction when their original archive can

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Downloaded wheels are removed after extraction when their original archive can
(pypi) Downloaded wheels are removed after extraction when their original archive can

Comment thread .bazelrc.deleted_packages
Comment on lines 47 to +48
common --deleted_packages=tests/integration/whl_library
common --deleted_packages=tests/integration/whl_storage

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could combine these 2 together. Having an extra test has a non-zero cost.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants