Release for security updates after certificates incident

Upgrade to rules_python 2.4.2 to pick up the certifi update, or follow the patch/workarounds in #4299 (comment)
-- rules_python maintainers

From https://discuss.python.org/t/information-for-python-users-regarding-emudhra-emsign-ca-certificate-security-incident/109576 :

A security incident affecting eMudhra intermediate certificates has required removing the emSign root CA certificate from CA bundles such as “certifi”. It’s important to immediately upgrade the software packages below to at least the versions specified and rebuild container images to fetch the latest system certificates.

Failure to do so could result in monster-in-the-middle (MITM) attacks against TLS clients, including python.org and pypi.org .

certifi 2026.10.19
pip 26.2.2
uv 0.13.4
sigstore 4.5.1

For all existing virtual environments on your system, please upgrade pip and uv to the latest versions.

In a Python project that uses Bazel, I find an older version of certifi (2026.7.22) in the MODULE.bazel.lock file:

  "facts": {
    "@@rules_python+//python/extensions:pip.bzl%pip": {
      "dist_hashes": {
        "https://pypi.org/simple": {
          "backports-tarfile": {
            "https://files.pythonhosted.org/packages/86/72/cd9b395f25e290e633655a100af28cb253e4393396264a98bd5f5951d50f/backports_tarfile-1.2.0.tar.gz": "sha256:d75e02c268746e1b8144c278978b6e98e85de6ad16f8e4b0844a154557eca991",
            "https://files.pythonhosted.org/packages/b9/fa/123043af240e49752f1c4bd24da5053b6bd00cad78c2be53c0d1e8b975bc/backports.tarfile-1.2.0-py3-none-any.whl": "sha256:77e284d754527b01fb1e6fa8a1afe577858ebe4e9dad8919e34c862cb399bc34"
          },
          "certifi": {
            "https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl": "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775",
            "https://files.pythonhosted.org/packages/a3/c2/24167ea9858356b47a87a50d39908bfdb72ceeefe0041586e704e5376b3a/certifi-2026.7.22.tar.gz": "sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55"
          },
....

But I am happy to see that the version update has already been applied in rules_python's main branch: #4297

Is there a new release planned to fix this security issue?

Or, if that release should take some more time, is there a way to use the already modified version, or to define a patch that Bazel can apply locally? Currently the dependency definition looks like this: bazel_dep(name = "rules_python", version = "2.4.2")

Thank you very much!

rules_python-2.4.2-certifi-2026-10-19.patch

Note

In rules_python 2.4.2, certifi==2026.7.22 only appears in
tools/publish/requirements_*.txt, which backs the rules_python_publish_deps
hub used by @rules_python//tools/publish:twine (the default twine_binary
for py_wheel's manual <name>.publish target).

Because rules_python's MODULE.bazel registers rules_python_publish_deps
with the pip extension, Bazel records certifi-2026.7.22 in
MODULE.bazel.lock under facts whenever the pip extension is evaluated,
even if @rules_python//tools/publish:twine is never fetched or built.

Workaround 1: Patch `tools/publish` via `single_version_override`

This workaround patches tools/publish/requirements_*.txt in…

Looking at what that PR fixes, it looks like the only thing of relevance to users is the tools/publish requirements files. That gets used by, I think its the twine binary used by default for the py_wheel .publish target.

As a user, you can specify your own twine binary as an arg. Alternatively you can patch that requirements file in tools/publish. I'll have a my agent post how to do both methods shortly.

#4301 for the patch release

build(deps): bump the uv group across 10 directories with 1 update (#…

rules_python-2.4.2-certifi-2026-10-19.patch

Note

In rules_python 2.4.2, certifi==2026.7.22 only appears in
tools/publish/requirements_*.txt, which backs the rules_python_publish_deps
hub used by @rules_python//tools/publish:twine (the default twine_binary
for py_wheel's manual <name>.publish target).

Because rules_python's MODULE.bazel registers rules_python_publish_deps
with the pip extension, Bazel records certifi-2026.7.22 in
MODULE.bazel.lock under facts whenever the pip extension is evaluated,
even if @rules_python//tools/publish:twine is never fetched or built.

Workaround 1: Patch `tools/publish` via `single_version_override`

This workaround patches tools/publish/requirements_*.txt inside rules_python
before the pip module extension runs. It both upgrades the default twine
environment and replaces certifi 2026.7.22 with certifi 2026.10.19 in
MODULE.bazel.lock.

Note

The full commit (711b314) from PR #4297
does not apply cleanly to the 2.4 release. The diff below contains the
relevant parts and applies cleanly to 2.4.2.

1. Create `patches/rules-python-certifi.patch`

Save patches/rules-python-certifi.patch (ensure patches/BUILD.bazel exists,
even if empty, so the label is valid):

diff --git a/tools/publish/requirements_darwin.txt b/tools/publish/requirements_darwin.txt
--- a/tools/publish/requirements_darwin.txt
+++ b/tools/publish/requirements_darwin.txt
@@ -8,5 +8,5 @@
     # via jaraco-context
-certifi==2026.7.22 \
-    --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \
-    --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55
+certifi==2026.10.19 \
+    --hash=sha256:035c1e85572b141bf1c3a3eff0e33a8eb74917522f2559075ff8d2fea780a1ae \
+    --hash=sha256:42d35340cc5b9ba78e818ddbff0c5b1c2b779de448a340fd1b989f5fa8670e1b
     # via requests
diff --git a/tools/publish/requirements_linux.txt b/tools/publish/requirements_linux.txt
--- a/tools/publish/requirements_linux.txt
+++ b/tools/publish/requirements_linux.txt
@@ -8,5 +8,5 @@
     # via jaraco-context
-certifi==2026.7.22 \
-    --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \
-    --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55
+certifi==2026.10.19 \
+    --hash=sha256:035c1e85572b141bf1c3a3eff0e33a8eb74917522f2559075ff8d2fea780a1ae \
+    --hash=sha256:42d35340cc5b9ba78e818ddbff0c5b1c2b779de448a340fd1b989f5fa8670e1b
     # via requests
diff --git a/tools/publish/requirements_universal.txt b/tools/publish/requirements_universal.txt
--- a/tools/publish/requirements_universal.txt
+++ b/tools/publish/requirements_universal.txt
@@ -8,5 +8,5 @@
     # via jaraco-context
-certifi==2026.7.22 \
-    --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \
-    --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55
+certifi==2026.10.19 \
+    --hash=sha256:035c1e85572b141bf1c3a3eff0e33a8eb74917522f2559075ff8d2fea780a1ae \
+    --hash=sha256:42d35340cc5b9ba78e818ddbff0c5b1c2b779de448a340fd1b989f5fa8670e1b
     # via requests
diff --git a/tools/publish/requirements_windows.txt b/tools/publish/requirements_windows.txt
--- a/tools/publish/requirements_windows.txt
+++ b/tools/publish/requirements_windows.txt
@@ -8,5 +8,5 @@
     # via jaraco-context
-certifi==2026.7.22 \
-    --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \
-    --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55
+certifi==2026.10.19 \
+    --hash=sha256:035c1e85572b141bf1c3a3eff0e33a8eb74917522f2559075ff8d2fea780a1ae \
+    --hash=sha256:42d35340cc5b9ba78e818ddbff0c5b1c2b779de448a340fd1b989f5fa8670e1b
     # via requests

2. Apply the patch in `MODULE.bazel`

bazel_dep(name = "rules_python", version = "2.4.2")
single_version_override(
    module_name = "rules_python",
    patch_strip = 1,
    patches = ["//patches:rules-python-certifi.patch"],
    version = "2.4.2",
)

3. Refresh `MODULE.bazel.lock` (if using a Bazel lockfile)

If your project tracks MODULE.bazel.lock, update it so the cached facts
reflect certifi 2026.10.19:

bazel mod deps --lockfile_mode=update

Workaround 2: Provide Your Own `twine` to `py_wheel`

If you use py_wheel and its <name>.publish target, you can supply twine
from your own locked pip.parse hub (where twine and certifi>=2026.10.19
are locked in your own requirements file) instead of using
@rules_python//tools/publish:twine.

Warning

In py_wheel,
twine_binary defaults to Label("//tools/publish:twine") when Bzlmod is
enabled, and if twine_binary: is checked before elif twine:. Therefore,
you must either override twine_binary directly, or set twine_binary = None
when passing twine.

Option A: Override `twine_binary` with `py_console_script_binary`

load("@rules_python//python:packaging.bzl", "py_wheel")
load(
    "@rules_python//python/entry_points:py_console_script_binary.bzl",
    "py_console_script_binary",
)

py_console_script_binary(
    name = "twine",
    pkg = "@pypi//twine",
    script = "twine",
)

py_wheel(
    name = "my_wheel",
    distribution = "my_pkg",
    twine_binary = ":twine",
    version = "0.1.0",
    deps = [":my_lib"],
)

Option B: Pass `twine` (`py_library`) and clear `twine_binary`

load("@rules_python//python:packaging.bzl", "py_wheel")

py_wheel(
    name = "my_wheel",
    distribution = "my_pkg",
    twine = "@pypi//twine",
    twine_binary = None,
    version = "0.1.0",
    deps = [":my_lib"],
)

(Note: Workaround 2 prevents @rules_python_publish_deps wheels from being
downloaded or used during builds/publishing, whereas Workaround 1 is required if
you also want certifi 2026.7.22 removed from MODULE.bazel.lock prior to the
patch release in #4301.)

Related to this, should we use @pypi//twine so that the override is easier to do? This is similar to what we did for pytest.

Our twine target is just a thin wrapper, right? Like just the entry point generation? If so, yeah, that would make sense. Though, we'd need some way to provide a default implementation. If @pypi goes back to the user's closure, and they don't have twine, then it'll give an error. Maybe thats OK. It basically pushes the problem of providing dependencies for our tools onto the user and we wouldn't have need to provide the various hard-coded urls for it.

@rickeylev

Hi @rickeylev !

If I understand correctly, only upgrading certifi leaves some gaps where rules_python still trusts the compromised certificates when it uses pip. Full disclosure: I'm not familiar with rules_python internals, and I used AI to trace through the code and run the tests below. I may be misreading how things fit together, so feel free to ignore this if it actually is nothing.

python/private/pypi/deps.bzl still seems to pin pypi__pip to pip 26.0.1. The discuss.python.org post quoted above lists pip 26.2.2 as a required upgrade. A CPython core developer explains why in the same thread:

pip always loads the CA certificates from its vendored certifi in addition to the system trust store, and there is no supported way to disable this. REQUESTS_CA_BUNDLE / --cert only add CAs; truststore is used “in addition to certifi” by design. As a result, any pip < 26.2.2 will keep trusting the compromised intermediate CAs regardless of the system trust store, so distrusting them at the OS level is not enough.

They also say that "Updating pip is the only reliable fix."
pip 26.0.1 vendors certifi 2026.01.04, which still contains emSign ECC Root CA - G3. So whenever rules_python's pinned pip makes a network request, it still trusts the compromised CAs, even when the host's trust store and the user's own certifi (including the one bumped in rules_python 2.4.3) are up to date.

If I understand correctly, rules_python's pip reaches the network in a few places:

compile_pip_requirements: pip-tools resolves and downloads through the pinned pip. This seems the most sensitive case, because with --generate-hashes the lockfile would record hashes of whatever was served.
Building sdists in whl_library: in 2.4.3, pip seems to build sdists with build isolation, so it downloads build dependencies (like setuptools) from the index itself. In a separate test with plain pip, those build-dependency downloads weren't hash-checked even in --require-hashes mode.
The pip fallback in pip.parse: when a distribution has no URL from the index metadata (for example via simpleapi_skip), whl_library seems to fall back to pip for the download itself.

I tried this with a small Bazel 9.3.1 workspace on rules_python 2.4.3, in a Debian trixie container whose OS trust store rejects the eMudhra test host (dv-g3b-emudhra.we-have-a-problem.com):

Case 1: --index-url pointing at the test host: pip-tools fetches the page over TLS and only skips it because of its Content-Type.
Case 2: building betterproto (sdist-only) logs "Installing build dependencies", so pip goes to the index itself. I couldn't point that index at the test host without also redirecting the Bazel downloader, so I'm assuming it behaves like the other two cases. extra_pip_args = ["--no-index", "--no-build-isolation"], may work as a mitigation for backends that are bundled.
Case 3: with six in simpleapi_skip and --index-url pointing at the test host: the TLS handshake succeeds, and pip only fails with "No matching distribution found".

With a single_version_override patch bumping pypi__pip to 26.2.2, cases 1 and 3 fail with CERTIFICATE_VERIFY_FAILED, as expected. Turns out, with pip 26.2.2, the pinned pip-tools 7.5.3 crashes in compile_pip_requirements. Bumping pypi__pip_tools to 7.6.2 fixed that, and resolution against PyPI worked again.

The Bazel downloader itself rejected the test host (PKIX path building failed), so the default wheel download path seems unaffected. This only seems to matter where rules_python's own pip goes to the network.

Would it makes sense to ship a fix that bumps pypi__pip to 26.2.2 (together with pypi__pip_tools 7.6.2) make sense? Users with hashed lockfiles are protected for the package files themselves, but I believe cases 1 and 2 aren't covered by hashes. For what it's worth, I haven't tested if the bumps would affect anything else. It seems like rules_python ships a Python 3.9 toolchain still and pip 26.2.2 requires Python >= 10, so there's a chance of some compatibility issues. No fixed 3.9 pip exists as far as I can tell, though.
Happy to share the test workspace or the patch.

Thanks for maintaining this!

Navigation Menu

Repository navigation

Release for security updates after certificates incident #4299

Workaround 1: Patch `tools/publish` via `single_version_override`

Workaround 1: Patch `tools/publish` via `single_version_override`

1. Create `patches/rules-python-certifi.patch`

2. Apply the patch in `MODULE.bazel`

3. Refresh `MODULE.bazel.lock` (if using a Bazel lockfile)

Workaround 2: Provide Your Own `twine` to `py_wheel`

Option A: Override `twine_binary` with `py_console_script_binary`

Option B: Pass `twine` (`py_library`) and clear `twine_binary`

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Release for security updates after certificates incident #4299

Description

Workaround 1: Patch tools/publish via single_version_override

Activity

rickeylev commented on Oct 22, 2026

rickeylev commented on Oct 22, 2026

Workaround 1: Patch tools/publish via single_version_override

1. Create patches/rules-python-certifi.patch

2. Apply the patch in MODULE.bazel

3. Refresh MODULE.bazel.lock (if using a Bazel lockfile)

Workaround 2: Provide Your Own twine to py_wheel

Option A: Override twine_binary with py_console_script_binary

Option B: Pass twine (py_library) and clear twine_binary

aignas commented on Oct 22, 2026

rickeylev commented on Oct 22, 2026

stephanretool commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Workaround 1: Patch `tools/publish` via `single_version_override`

Workaround 1: Patch `tools/publish` via `single_version_override`

1. Create `patches/rules-python-certifi.patch`

2. Apply the patch in `MODULE.bazel`

3. Refresh `MODULE.bazel.lock` (if using a Bazel lockfile)

Workaround 2: Provide Your Own `twine` to `py_wheel`

Option A: Override `twine_binary` with `py_console_script_binary`

Option B: Pass `twine` (`py_library`) and clear `twine_binary`