Skip to content

Transition OSS Python dependencies and CI builds to hermetic Python. - #1566

Merged
copybara-service[bot] merged 1 commit into
masterfrom
test_997953043
Oct 23, 2026
Merged

copybara-service[bot] merged 1 commit into
masterfrom
test_997953043

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Transition OSS Python dependencies and CI builds to hermetic Python.

Open-source builds and CI pipelines previously relied on legacy @pypi_<pkg>//:pkg repository targets, @release_or_nightly workspace rules, and host-installed Python packages (pip install) before invoking Bazel. With the migration to rules_python hermetic Python toolchains, Bazel resolves Python interpreter versions and wheel dependencies directly via requirements.txt lockfiles and @pypi//<pkg>:pkg targets, making host-level package installation and legacy workspace dependencies redundant and error-prone.

Migrate OSS Python dependency rewrites and oss_scripts/pip_package/BUILD targets to @pypi//<pkg>:pkg, remove the obsolete extra_py_deps helper and @org_tensorflow_datasets workspace archive, and declare explicit test dependencies in BUILD targets. Every //third_party/py/absl... dependency maps to the same @pypi//absl_py:pkg, so Copybara keeps the first one in each deps list and reversibly comments out the rest; BUILD files can list all the absl deps they use. Update oss_scripts/configure.sh and py_deps_profile.bzl to persist HERMETIC_PYTHON_VERSION, IS_NIGHTLY, and TF_VERSION into .bazelrc, re-evaluate the repository rule when environment variables change, and pin tensorflow==<TF_VERSION> when re-locking requirements. configure.sh does not re-lock when the lock file already has the requested TF_VERSION, so the committed lock file is not replaced with stale tf-keras or pre-release versions. Presubmit, release, and Docker scripts pass HERMETIC_PYTHON_VERSION instead of managing host virtual environments.

Presubmits use the release lock file by default. Nightly is used only when IS_NIGHTLY=nightly is set (the continuous integration configs set it) or TF_VERSION is a dev version. prepare_tf_dep.sh updates setup.nightly.py for nightly builds and setup.py otherwise, the same file build_pip_package uses. On macOS and Windows, Bazel is installed (source oss_scripts/install_bazel.sh) before configure.sh runs bazel run; configure.sh --fetch_bazel_config_only downloads .bazelversion first.

Additionally, remove the rules_ml_toolchain override from WORKSPACE so the newer version from TF's tf_workspace3() is used, build new deps lists instead of using deps += on default arguments in tftext.bzl.oss, and read only the first line of .bazelversion in run_tests.sh.

@copybara-service
copybara-service Bot force-pushed the test_997953043 branch 16 times, most recently from 7cb4917 to a599cde Compare October 16, 2026 14:36
@copybara-service
copybara-service Bot force-pushed the test_997953043 branch 5 times, most recently from 50fa054 to f34294c Compare October 23, 2026 15:53
Open-source builds and CI pipelines previously relied on legacy `@pypi_<pkg>//:pkg` repository targets, `@release_or_nightly` workspace rules, and host-installed Python packages (`pip install`) before invoking Bazel. With the migration to `rules_python` hermetic Python toolchains, Bazel resolves Python interpreter versions and wheel dependencies directly via `requirements.txt` lockfiles and `@pypi//<pkg>:pkg` targets, making host-level package installation and legacy workspace dependencies redundant and error-prone.

Migrate OSS Python dependency rewrites and `oss_scripts/pip_package/BUILD` targets to `@pypi//<pkg>:pkg`, remove the obsolete `extra_py_deps` helper and `@org_tensorflow_datasets` workspace archive, and declare explicit test dependencies in `BUILD` targets. Every `//third_party/py/absl...` dependency maps to the same `@pypi//absl_py:pkg`, so Copybara keeps the first one in each `deps` list and reversibly comments out the rest; `BUILD` files can list all the absl deps they use. Update `oss_scripts/configure.sh` and `py_deps_profile.bzl` to persist `HERMETIC_PYTHON_VERSION`, `IS_NIGHTLY`, and `TF_VERSION` into `.bazelrc`, re-evaluate the repository rule when environment variables change, and pin `tensorflow==<TF_VERSION>` when re-locking requirements. `configure.sh` does not re-lock when the lock file already has the requested `TF_VERSION`, so the committed lock file is not replaced with stale `tf-keras` or pre-release versions. Presubmit, release, and Docker scripts pass `HERMETIC_PYTHON_VERSION` instead of managing host virtual environments.

Presubmits use the release lock file by default. Nightly is used only when `IS_NIGHTLY=nightly` is set (the continuous integration configs set it) or `TF_VERSION` is a dev version. `prepare_tf_dep.sh` updates `setup.nightly.py` for nightly builds and `setup.py` otherwise, the same file `build_pip_package` uses. On macOS and Windows, Bazel is installed (`source oss_scripts/install_bazel.sh`) before `configure.sh` runs `bazel run`; `configure.sh --fetch_bazel_config_only` downloads `.bazelversion` first.

Additionally, remove the `rules_ml_toolchain` override from `WORKSPACE` so the newer version from TF's `tf_workspace3()` is used, build new `deps` lists instead of using `deps +=` on default arguments in `tftext.bzl.oss`, and read only the first line of `.bazelversion` in `run_tests.sh`.

PiperOrigin-RevId: 1005644968
@copybara-service
copybara-service Bot merged commit d38d2f4 into master Oct 23, 2026
4 checks passed
@copybara-service
copybara-service Bot deleted the test_997953043 branch October 23, 2026 16:09

This branch was successfully deployed

1 active deployment
github-pages — d38d2f40 Deployed Oct 23, 2026 by github-pages[bot] via deploy #339
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant