Update lib/mbedtls from 2.28.3 to 4.2.0

[dhalbert]

• Fixes Update lib/mbedtls submodule from 2.28.3 to 3.6.7 or 4.2.0 #11149
• Fixes TLS connections with local CA fails on Pico 2 W (RP2350 + CYW43439) with MBEDTLS_ERR_X509_CERT_VERIFY_FAILED #10339 (fixes a bug) (@davebarkerxyz)
• Fixes Should we update Mbedtls submodule ref? #10738 (@samblenny)

🤖 Generated with Claude Code

This comment was originally written by Claude and then edited by @dhalbert.

Updates lib/mbedtls from 2.28.3 (March 2023) to 4.2.0. Used only by raspberrypi; espressif has its own copy, at 4.0.0.

Changes in mbedtls 4.0

details

mbedtls 4.0 moved crypto into a nested tf-psa-crypto submodule and split the configuration in two: MBEDTLS_CONFIG_FILE now covers only TLS and X.509, while algorithms are requested with PSA_WANT_* from TF_PSA_CRYPTO_CONFIG_FILE. The legacy mbedtls_sha256_*() headers became private, and TLS is now built on PSA rather than the legacy crypto API.

• tools/ci_fetch_deps.py initializes tf-psa-crypto and framework by name rather than with --recursive, which would also pull tf-psa-crypto's own framework copy and drivers/pqcp/mldsa-native. Keyed off the checkout rather than the requested paths, because the targets that pull lib/mbedtls name it three different ways. This also covers make fetch-all-submodules and make fetch-port-submodules, which both shell out to this script.
• ports/raspberrypi/Makefile generates the PSA driver-wrapper dispatch layer from tf-psa-crypto's jinja templates (~0.1 s, once per build directory), adding jsonschema to requirements-dev.txt; jinja2 was already there. Source directories are globbed rather than listed, since every file is internally guarded by its config macros and we build with -ffunction-sections/--gc-sections.
• Randomness comes from the port TRNG via MBEDTLS_PSA_CRYPTO_EXTERNAL_RNG, since 4.x dropped MBEDTLS_ENTROPY_HARDWARE_ALT. This also keeps entropy.c and ctr_drbg.c out of the build.
• MBEDTLS_HAVE_TIME is gone: nothing we enable consumes time, so mbedtls_ms_time() and rp2_rtctime_seconds() are no longer needed. MBEDTLS_HAVE_TIME_DATE stays off, as before and as on espressif — CircuitPython does not know the wall clock unless the program sets it, so checking certificate dates would reject valid certificates rather than catch expired ones.
• hashlib now uses the PSA hash API on every port, so shared-module/hashlib/ has no version branches left. Ports with CIRCUITPY_HASHLIB_MBEDTLS_ONLY build a minimal PSA core; MICROPY_TRACKED_ALLOC is enabled for them because tf-psa-crypto's platform.c gets linked.
• Removes the MBEDTLS_VERSION_MAJOR == 3 branches, which no port ever compiled.
• lib/mbedtls_errors/mp_mbedtls_errors.c regenerated; do-mp.sh updated for generate_errors.pl's new four-argument signature and to strip the trailing whitespace the generator emits, so re-running it reproduces exactly what is checked in.

Config settings stand alone, and do not modify a set of defaults

Note that MBEDTLS_CONFIG_FILE and TF_PSA_CRYPTO_CONFIG_FILE replace the config headers from the upstream instead of overlaying them, so we must be careful to be explicit and set every config option we want. (ESP-IDF avoids this for crypto by using TF_PSA_CRYPTO_USER_CONFIG_FILE (overlay) and only replacing the TLS config; MicroPython overlays too. Overlaying was considered and rejected here: it would mean inheriting 112 enabled options and writing ~70 #undefs, which is more to maintain and silently inherits whatever upstream adds next — untenable on a 2 MB board.)

Flash size increase

Board	before	after	delta
raspberry_pi_pico_w	1515468 B (96.60%)	1537348 B (98.00%)	+21880 B
raspberry_pi_pico (hashlib changes only)	972432 B (93.10%)	971700 B (93.03%)	−732 B
feather_nrf52840_express (hashlib changes only)	659908 B (82.62%)	659948 B (82.63%)	+40 B
raspberry_pi_pico2_w	not measured	1507376 B (96.09%)	—

TLS 1.3 is available in 4.x but left off, matching espressif: it cost 25648 B on Pico W, and espressif does not set CONFIG_MBEDTLS_SSL_PROTO_TLS1_3 either. mbedtls_config.h records what to re-enable if that changes.

Differences from mbedtls options used in espressif

details

So reviewers do not read these as oversights:

• ChaCha20-Poly1305 is enabled in raspberrypi and not in espressif. RP2 has no AES accelerator and we build with MBEDTLS_AES_ROM_TABLES. Measured on a 604 kB HTTPS transfer with the ciphersuite pinned: Pico W 176.2 vs 122.2 kB/s (+44%), Pico 2 W 345.9 vs 205.5 kB/s (+68%). These transfers are CPU-bound on the bulk cipher, not on WiFi — if they were network-bound both ciphers would measure the same. Many Internet hosts including Cloudfront negotiate ChaCha20 when both are offered, so this will improve . Costs 6192 B.
• P-521, Brainpool and secp256k1 off. No public CA issues from them. P-521 alone cost 5808 B.
• ALPN and MAX_FRAGMENT_LENGTH off — upstream defaults, but CircuitPython's ssl module exposes no API to set either.
• DTLS off — common_hal_ssl_sslcontext_wrap_socket() rejects anything that is not SOCKETPOOL_SOCK_STREAM, so it is unreachable.
• Session tickets off — would reintroduce the MBEDTLS_HAVE_TIME dependency, and a fresh context is built per connection anyway.

Fixed an existing entropy bug, discovered in passing

• mbedtls_hardware_poll() in lib/mbedtls_config/mbedtls_port.c passed data to common_hal_os_urandom() instead of output. This is now changed to use mbedtls_psa_external_get_random() with the correct arguments.

Testing

• Tested TLS by hostname, by IP address (the TLS connections with local CA fails on Pico 2 W (RP2350 + CYW43439) with MBEDTLS_ERR_X509_CERT_VERIFY_FAILED #10339 fix), with explicit cadata, and through the bundled CA store. Checked against two Internet servers plus a local server.
• hashlib tested for correct computations
• Tested that espressif HTTPS fetch unaffected.
• Cipher benchmarks on both W boards, as above.

Not covered by hardware testing: ECDSA server certificates (every test chain was RSA), client certificates, and hashlib on nordic specifically. An audit of the derived MBEDTLS_PSA_BUILTIN_* capabilities against every psa_* call reachable from the TLS and X.509 paths shows the required builtins are all present for those cases, but they have not been exercised.

[dhalbert]

I requested Claude record this build-process design decision.

Note for the record: why the PSA driver wrappers are generated at build time rather than checked in

mbedtls 4.x dispatches PSA crypto through psa_crypto_driver_wrappers.h and
psa_crypto_driver_wrappers_no_static.c, which upstream generates from jinja templates
instead of shipping. This PR generates them during the build. Checking them in was
considered and deliberately not done; recording the reasoning here since it is the kind
of thing that gets re-litigated at the next bump.

Relevant facts:

• The output is a fixed artifact. generate_driver_wrappers.py takes only an output
  directory — no config input — and produces byte-identical files for every port and
  configuration. Verified: the raspberry_pi_pico_w build's copy has the same sha256 as
  a standalone run.
• Its inputs are exactly three files, per upstream's own make rule
  (tf-psa-crypto/core/crypto-library.make:18-20): the generator plus its two templates.
• The two files total about 136 kB of generated source.
• Generation costs ~0.1 s, once per build directory, and only re-runs when the generator
  or templates change.

Arguments for checking them in: ordinary builds would run no Python codegen at all, so a
nordic build — a port with nothing to do with TLS — would not need jinja2 and
jsonschema present at build time. It would also collapse the codegen rule, which
currently appears in both ports/raspberrypi/Makefile and py/circuitpy_defns.mk.

Why build-time generation won anyway:

• It matches upstream CMake and ESP-IDF, so the wrappers can never be stale with respect
  to the submodule.
• jsonschema and the framework submodule have to stay available regardless, because a
  regeneration target would need them. Checking the files in therefore does not remove
  either dependency; it only moves the point at which they are required.
• The staleness failure mode is silent and would land on whoever bumps mbedtls next.
  Four config regressions in this PR were of exactly that shape — invisible to compiling
  and linking — so the bias is toward mechanisms that cannot go stale.

If this is revisited, the shape that addresses the staleness objection without requiring
Python for ordinary builds is: commit the two files plus a stamp containing the sha256 of
the three generator inputs, have the build compare it with sha256sum (no Python), and
fail with a message pointing at a make mbedtls-regenerate-psa-wrappers target. That
would also give lib/mbedtls_errors/mp_mbedtls_errors.c a freshness check, which it
currently does not have.

[tannewt]

One weird comment. Looks great otherwise! Love seeing all of the v3 deletions.

[tannewt]

What does this have to do with the IDF?

[dhalbert]

I put back and clarified the verbose comment Claude had added:

    // For raspberrypi, we must free PSA crypto, because there are GC-heap objects
    // in the key slots.  We can't put this call in ssl_reset() because that's a
    // shared-module implementation. Unlike raspberrypi, espressif ESP-IDF inits PSA
    // once at boot and would never re-init it.
    // common_hal_ssl_sslcontext_construct() re-inits PSA on demand.
    // So for raspberrypi, we must call mbedtls_psa_crypto_free() explicitly.
    mbedtls_psa_crypto_free();