Resolve the wolfSSL FIPS Ready bundle at CI run time

.github/workflows/README.md

@@ -30,7 +30,7 @@ ready_for_review) and on every push to `master`, `main`, or
 | `simple.yml` | Builds wolfProvider against the matrix of supported wolfSSL + OpenSSL refs and runs `make check`. The baseline "did anything obvious break" check. |
 | `smoke-test.yml` | Minimal end-to-end: build, load the provider into stock OpenSSL, run `openssl list -providers` and a handful of `openssl` subcommands. Catches link-time and provider-registration regressions. |
 | `cmdline.yml` | Runs `scripts/cmd_test/do-cmd-tests.sh` — exercises every `openssl` CLI verb (genrsa, pkeyutl, enc, dgst, …) through wolfProvider. |
-| `fips-ready.yml` | Same as `simple` but builds wolfSSL with `--enable-fips=ready`. Sanity check that FIPS-ready compiles and basic tests pass without the full FIPS bundle. |
+| `fips-ready.yml` | Downloads FIPS Ready bundles from wolfssl.com (via `scripts/fetch-fips-ready.sh`) and builds wolfSSL with `--enable-fips=ready` against each. Matrix resolved at run time from `_discover-versions.yml`'s `fips_ready_pr_array`: the back-compat anchors plus the newest bundle. `ready` is not a fixed FIPS version: v7 through bundle 5.9.1, v8 from 5.9.2 on. |
 | `seed-src.yml` | Builds with `--enable-seed-src` (entropy seed source variant) and `-DWP_TEST_SECCOMP_SANDBOX`, then runs the unit tests including the OpenSSH fork+seccomp-sandbox regression suite. |
 | `multi-compiler.yml` | Cross-compiler sweep: gcc-9 through gcc-14 and clang-12 through latest. Catches toolchain-specific warnings / UB. |
 | `codespell.yml` | Spell-check on tracked source. `*.patch` is excluded because OSP patches mirror upstream source whose original spelling we shouldn't silently rewrite. |
@@ -59,6 +59,11 @@ push and dominate the merge queue. Nightly is the right cadence for
 catching regressions in third-party integration that wouldn't show up
 in our unit tests.
 
+`nightly-fips-ready.yml` rides along out-of-wave, running the same steps
+as `fips-ready.yml` over `fips_ready_all_array`: every FIPS Ready bundle
+hosted at or above the 5.8.2 floor, restoring the versions the PR set
+leaves out.
+
 ### Running a nightly job on a PR (label toggles)
 
 `pr-osp-select.yml` lets you pull any nightly job into a PR on demand —
@@ -69,7 +74,7 @@ per-app workflows stay untouched.
 | Label | Effect |
 |-------|--------|
 | `ci:<name>` | Run that one job (e.g. `ci:hostap`, `ci:curl`, `ci:static-analysis`). Add several to run several. |
-| `ci:all` | Run the whole fan-out (all 43 jobs). |
+| `ci:all` | Run the whole fan-out (all 44 jobs). |
 | (no label) | Nothing runs — a normal PR is unaffected. |
 
 `<name>` is the job key in the table below (the workflow base name, e.g.
@@ -96,8 +101,9 @@ gh workflow run pr-osp-select.yml --ref <branch> -f jobs="all"
 
 ### What runs in the nightly fan-out
 
-43 workflows total: 40 third-party OSS integrations, 2 internal
-validations, and the static-analysis suite. Every one of these patches
+44 workflows total: 40 third-party OSS integrations, 3 internal
+validations (including nightly-fips-ready.yml, reachable via
+`ci:fips-ready`), and the static-analysis suite. Every one of these patches
 the upstream project (where needed) via `osp/wolfProvider/<app>/*.patch`
 from [wolfssl/osp](https://github.com/wolfssl/osp), builds it against
 the replace-default wolfProvider `.deb` stack, and runs the project's

.github/workflows/_discover-versions.yml

@@ -30,6 +30,21 @@ on:
       openssl_all_releases_array:
         description: 'JSON array of every upstream openssl-3.X.Y release tag, sorted ascending. Used by openssl-version.yml so the sweep tracks upstream automatically.'
         value: ${{ jobs.discover.outputs.openssl_all_releases_array }}
+      fips_ready_latest:
+        description: 'Plain string, newest hosted FIPS Ready bundle version (e.g. 5.9.2)'
+        value: ${{ jobs.discover.outputs.fips_ready_latest }}
+      fips_ready_pr_array:
+        description: 'JSON array of FIPS Ready bundle versions for PR runs: the back-compat anchors plus the newest'
+        value: ${{ jobs.discover.outputs.fips_ready_pr_array }}
+      fips_ready_all_array:
+        description: 'JSON array of every hosted FIPS Ready bundle version at or above the floor. Used by the nightly sweep.'
+        value: ${{ jobs.discover.outputs.fips_ready_all_array }}
+      fips_ready_resolved:
+        description: '"true" if FIPS Ready bundle discovery succeeded, "false" if the outputs above are a degraded floor-only fallback. FIPS-specific callers must fail on "false" rather than silently testing a stale matrix.'
+        value: ${{ jobs.discover.outputs.fips_ready_resolved }}
+      fips_ready_version_map:
+        description: 'JSON object mapping each FIPS Ready bundle version to the FIPS module (v5/v7/v8) that --enable-fips=ready actually resolves to for that bundle.'
+        value: ${{ jobs.discover.outputs.fips_ready_version_map }}
 
 jobs:
   discover:
@@ -49,7 +64,17 @@ jobs:
       openssl_latest_ref: ${{ steps.resolve.outputs.openssl_latest_ref }}
       openssl_latest_ref_array: ${{ steps.resolve.outputs.openssl_latest_ref_array }}
       openssl_all_releases_array: ${{ steps.resolve.outputs.openssl_all_releases_array }}
+      fips_ready_latest: ${{ steps.resolve.outputs.fips_ready_latest }}
+      fips_ready_pr_array: ${{ steps.resolve.outputs.fips_ready_pr_array }}
+      fips_ready_all_array: ${{ steps.resolve.outputs.fips_ready_all_array }}
+      fips_ready_resolved: ${{ steps.resolve.outputs.fips_ready_resolved }}
+      fips_ready_version_map: ${{ steps.resolve.outputs.fips_ready_version_map }}
     steps:
+      - name: Checkout wolfProvider
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
       - name: Install ORAS (best-effort)
         run: |
           set -uo pipefail
@@ -169,11 +194,61 @@ jobs:
           OSSL_ALL_JSON=$(printf '%s\n' "$OSSL_ALL" | jq -R . | jq -s -c .)
           OSSL_LATEST=$(echo "$OSSL_ALL" | tail -n 1)
 
+          # FIPS Ready: every bundle hosted on wolfssl.com at or above the floor.
+          # Floor 5.8.2: the oldest bundle wolfProvider is tested against.
+          FR_FLOOR="5.8.2"
+          FR_OK=true
+          # Bound the probe so a hung wolfssl.com cannot eat this shared job's
+          # timeout and take all ~40 callers down; a timeout degrades to FR_OK=false.
+          FR_ALL=$(timeout 240 ./scripts/fetch-fips-ready.sh --list --floor "$FR_FLOOR") || true
+          if [ -z "${FR_ALL:-}" ]; then
+            # ~40 workflows share this job for wolfssl/openssl outputs only,
+            # so a FIPS resolution failure must not fail the whole job. FR_OK
+            # lets the FIPS-specific callers fail loudly on their own instead
+            # of reporting green on a stale, near-empty fallback matrix.
+            echo "::warning::Could not list FIPS Ready bundles; falling back to $FR_FLOOR (FIPS callers will fail this run)"
+            FR_ALL="$FR_FLOOR"
+            FR_OK=false
+          fi
+          FR_ALL_JSON=$(printf '%s\n' "$FR_ALL" | jq -R . | jq -s -c .)
+          FR_LATEST=$(echo "$FR_ALL" | tail -n 1)
+
+          # `ready` does not name a fixed FIPS module; map each hosted bundle
+          # to the module it actually builds (from wolfSSL's configure.ac
+          # `ready|vN-ready` case, checked per tag). Extend the boundaries
+          # below when a new FIPS generation ships.
+          fips_module_for() {
+            local key
+            key=$(printf '%s' "$1" | awk -F. '{ printf "%05d.%05d.%05d", $1, $2, $3 }')
+            if   [[ "$key" > "00005.00009.00001" ]]; then echo "v8"  # > 5.9.1
+            elif [[ "$key" > "00005.00007.00000" ]]; then echo "v7"  # > 5.7.0
+            else echo "v5"
+            fi
+          }
+          FR_MAP_JSON="{}"
+          while IFS= read -r ver; do
+            FR_MAP_JSON=$(jq -c --arg v "$ver" --arg f "$(fips_module_for "$ver")" \
+              '. + {($v): $f}' <<< "$FR_MAP_JSON")
+          done <<< "$FR_ALL"
+          # 5.8.4 and 5.9.1 are the back-compat anchors: the last two bundles
+          # where --enable-fips=ready still meant FIPS v7, which 5.9.2 moved to v8.
+          # Intersected with the hosted set so a delisted anchor drops out.
+          FR_PR_JSON=$(printf '%s\n' "5.8.4" "5.9.1" "$FR_LATEST" | jq -R . \
+            | jq -s -c --argjson all "$FR_ALL_JSON" \
+                '[.[] | select(. as $v | $all | index($v))] | unique_by(split(".")|map(tonumber))')
+          if [ "$FR_PR_JSON" = "[]" ]; then
+            FR_PR_JSON="[\"$FR_LATEST\"]"
+          fi
+
           echo "wolfSSL .deb ref (actual ghcr deb):  $WOLFSSL_DEB_REF"
           echo "wolfSSL upstream latest -stable:    $WOLFSSL_LATEST"
           echo "OpenSSL Bookworm stock:             openssl-$OSSL (raw: $OSSL_RAW)"
           echo "OpenSSL upstream latest:            $OSSL_LATEST"
           echo "OpenSSL upstream releases tracked:  $(echo "$OSSL_ALL" | wc -l) tags"
+          echo "FIPS Ready latest bundle:           $FR_LATEST"
+          echo "FIPS Ready PR set:                  $FR_PR_JSON"
+          echo "FIPS Ready full set (floor $FR_FLOOR):  $FR_ALL_JSON"
+          echo "FIPS Ready version map:             $FR_MAP_JSON"
 
           {
             echo "wolfssl_ref=$WOLFSSL_DEB_REF"
@@ -186,4 +261,9 @@ jobs:
             echo "openssl_latest_ref=$OSSL_LATEST"
             echo "openssl_latest_ref_array=[\"$OSSL_LATEST\"]"
             echo "openssl_all_releases_array=$OSSL_ALL_JSON"
+            echo "fips_ready_latest=$FR_LATEST"
+            echo "fips_ready_pr_array=$FR_PR_JSON"
+            echo "fips_ready_all_array=$FR_ALL_JSON"
+            echo "fips_ready_resolved=$FR_OK"
+            echo "fips_ready_version_map=$FR_MAP_JSON"
           } >> "$GITHUB_OUTPUT"

.github/workflows/fips-ready.yml

@@ -25,23 +25,53 @@ concurrency:
 # END OF COMMON SECTION
 
 jobs:
+  test_fetch_script:
+    if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
+    name: Test fetch-fips-ready.sh
+    runs-on: ubuntu-22.04
+    timeout-minutes: 5
+    steps:
+      - name: Checkout wolfProvider
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+      - name: Run network-free resolver tests
+        run: ./scripts/test-fetch-fips-ready.sh
+
   discover_versions:
+    needs: test_fetch_script
     if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
     uses: ./.github/workflows/_discover-versions.yml
 
-  fips_ready_test:
+  # A resolution failure makes discover_versions fall back to testing only
+  # the floor version. Fine for the ~40 unrelated workflows sharing that
+  # job, but here it would report green while skipping most of the matrix.
+  require_fips_resolved:
     needs: discover_versions
     if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
-    name: FIPS Ready Bundle Test
+    name: Require FIPS Ready bundle discovery succeeded
+    runs-on: ubuntu-22.04
+    timeout-minutes: 2
+    steps:
+      - name: Fail if discovery fell back to the floor
+        if: needs.discover_versions.outputs.fips_ready_resolved != 'true'
+        run: |
+          echo "::error::FIPS Ready bundle discovery failed and fell back to a single floor version; refusing to report success on a silently degraded matrix. See the discover_versions job log."
+          exit 1
+
+  fips_ready_test:
+    needs: [discover_versions, require_fips_resolved]
+    if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
+    name: FIPS Ready ${{ matrix.wolfssl_bundle_ref }} (FIPS ${{ fromJson(needs.discover_versions.outputs.fips_ready_version_map)[matrix.wolfssl_bundle_ref] }}, ${{ matrix.openssl_ref }})
     runs-on: ubuntu-22.04
     permissions:
       contents: read
       packages: write
-    timeout-minutes: 20
+    timeout-minutes: 30
     strategy:
       fail-fast: false
       matrix:
-        wolfssl_bundle_ref: [ '5.8.2' ]
+        wolfssl_bundle_ref: ${{ fromJson(needs.discover_versions.outputs.fips_ready_pr_array) }}
         openssl_ref: ${{ fromJson(needs.discover_versions.outputs.openssl_latest_ref_array) }}
         # force_fail collapsed into sequential runs in the test step
     steps:
@@ -65,21 +95,8 @@ jobs:
 
       - name: Download FIPS Ready Bundle
         run: |
-          # Download FIPS ready bundle from wolfSSL website
-          BUNDLE_URL="https://www.wolfssl.com/wolfssl-${{matrix.wolfssl_bundle_ref}}-gplv3-fips-ready.zip"
-
-          wget -O wolfssl-fips-ready.zip "$BUNDLE_URL"
-          unzip wolfssl-fips-ready.zip
-
-          # Find the extracted directory (build script requires directory, not zip)
-          BUNDLE_DIR=$(find . -maxdepth 1 -type d -name "*fips-ready*" | head -n 1)
-          if [ -z "$BUNDLE_DIR" ]; then
-            echo "ERROR: Could not find FIPS ready bundle directory after extraction"
-            ls -la
-            exit 1
-          fi
-
-          echo "FIPS_BUNDLE_PATH=$(pwd)/$BUNDLE_DIR" >> $GITHUB_ENV
+          BUNDLE_DIR=$(./scripts/fetch-fips-ready.sh "${{ matrix.wolfssl_bundle_ref }}")
+          echo "FIPS_BUNDLE_PATH=$BUNDLE_DIR" >> $GITHUB_ENV
           echo "Found FIPS bundle directory at: $BUNDLE_DIR"
 
       - name: Build wolfProvider with FIPS Ready Bundle

.github/workflows/nightly-fips-ready.yml

@@ -0,0 +1,117 @@
+name: Nightly FIPS Ready Bundle Tests
+
+# Sweeps EVERY hosted FIPS Ready bundle at or above the floor. PR-time
+# fips-ready.yml runs the narrower fips_ready_pr_array (the back-compat
+# anchors plus the newest) to keep PR turnaround short; this restores the
+# versions it drops. Both sets are resolved at run time by
+# _discover-versions.yml, so a new FIPS Ready release joins the sweep on its
+# own.
+
+on:
+  workflow_call: {}
+  workflow_dispatch: {}
+
+# No concurrency group: github.workflow is the caller's here, so any group collides with sibling calls.
+
+jobs:
+  test_fetch_script:
+    name: Test fetch-fips-ready.sh
+    runs-on: ubuntu-22.04
+    timeout-minutes: 5
+    steps:
+      - name: Checkout wolfProvider
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+      - name: Run network-free resolver tests
+        run: ./scripts/test-fetch-fips-ready.sh
+
+  discover_versions:
+    needs: test_fetch_script
+    uses: ./.github/workflows/_discover-versions.yml
+
+  # See fips-ready.yml for why this can't be left to a silent floor fallback.
+  require_fips_resolved:
+    needs: discover_versions
+    name: Require FIPS Ready bundle discovery succeeded
+    runs-on: ubuntu-22.04
+    timeout-minutes: 2
+    steps:
+      - name: Fail if discovery fell back to the floor
+        if: needs.discover_versions.outputs.fips_ready_resolved != 'true'
+        run: |
+          echo "::error::FIPS Ready bundle discovery failed and fell back to a single floor version; refusing to report success on a silently degraded matrix. See the discover_versions job log."
+          exit 1
+
+  fips_ready_test:
+    needs: [discover_versions, require_fips_resolved]
+    name: Nightly FIPS Ready ${{ matrix.wolfssl_bundle_ref }} (FIPS ${{ fromJson(needs.discover_versions.outputs.fips_ready_version_map)[matrix.wolfssl_bundle_ref] }}, ${{ matrix.openssl_ref }})
+    runs-on: ubuntu-22.04
+    permissions:
+      contents: read
+      packages: write
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      matrix:
+        wolfssl_bundle_ref: ${{ fromJson(needs.discover_versions.outputs.fips_ready_all_array) }}
+        openssl_ref: ${{ fromJson(needs.discover_versions.outputs.openssl_latest_ref_array) }}
+    steps:
+      - name: Checkout wolfProvider
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Cache build dependencies
+        id: deps
+        uses: ./.github/actions/oras-build-deps
+        with:
+          variant: fips-ready
+          openssl_ref: ${{ matrix.openssl_ref }}
+          wolfssl_fixed_key: wolfssl-fips-${{ matrix.wolfssl_bundle_ref }}-${{ matrix.openssl_ref }}
+          # The build below sets WOLFPROV_FIPS_FORCE_FAIL, which adds
+          # -DHAVE_FORCE_FIPS_FAILURE to the wolfSSL build; key the cached
+          # install on it so a wolfSSL built without the define is never reused.
+          extra_key: forcefail
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Download FIPS Ready Bundle
+        run: |
+          BUNDLE_DIR=$(./scripts/fetch-fips-ready.sh "${{ matrix.wolfssl_bundle_ref }}")
+          echo "FIPS_BUNDLE_PATH=$BUNDLE_DIR" >> $GITHUB_ENV
+          echo "Found FIPS bundle directory at: $BUNDLE_DIR"
+
+      - name: Build wolfProvider with FIPS Ready Bundle
+        run: |
+          # WOLFPROV_FIPS_FORCE_FAIL is an internal, test-only knob (no CLI
+          # flag) that enables HAVE_FORCE_FIPS_FAILURE for the fips_status test.
+          OPENSSL_TAG=${{ matrix.openssl_ref }} \
+          WOLFPROV_FIPS_FORCE_FAIL=1 \
+          ./scripts/build-wolfprovider.sh --fips-bundle="$FIPS_BUNDLE_PATH" \
+          --fips-check=ready --wolfssl-ver=v${{matrix.wolfssl_bundle_ref}}-stable
+
+      - name: Push build dependencies
+        uses: ./.github/actions/oras-build-deps-push
+        with:
+          registry: ${{ steps.deps.outputs.registry }}
+          openssl_install_tag: ${{ steps.deps.outputs.openssl_install_tag }}
+          wolfssl_install_tag: ${{ steps.deps.outputs.wolfssl_install_tag }}
+          openssl_hit: ${{ steps.deps.outputs.openssl_hit }}
+          wolfssl_hit: ${{ steps.deps.outputs.wolfssl_hit }}
+
+      - name: Run FIPS Command Tests
+        run: |
+          export WOLFSSL_ISFIPS=1
+          source scripts/env-setup
+
+          # --- normal mode ---
+          ./scripts/cmd_test/do-cmd-tests.sh
+
+          # --- force-fail mode ---
+          WOLFPROV_FORCE_FAIL=1 ./scripts/cmd_test/do-cmd-tests.sh
+
+      - name: Run FIPS Status Test
+        run: |
+          export WOLFSSL_ISFIPS=1
+          source scripts/env-setup
+          ./test/standalone/tests/fips_status/run.sh

.github/workflows/nightly-osp.yml

@@ -4,8 +4,8 @@ name: Nightly OSP Suite
 # wolfssl stable). Stays dynamic across version bumps.
 # Wave 2 = v5.8.4-stable pinned (back-compat line), gated on wave1-done
 # with if: always() so a single Wave 1 failure doesn't skip the
-# older-line coverage. multi-compiler and static-analysis run once
-# outside the waves.
+# older-line coverage. multi-compiler, fips-ready and static-analysis run
+# once outside the waves.
 # nginx-pqc is Wave 1 only: PQC has a v5.9.2-stable floor, so the
 # v5.8.4-stable Wave 2 line has no eligible ref to run.
 
@@ -169,3 +169,4 @@ jobs:
   # === Out-of-wave: not wolfssl-version-split ===
   static-analysis: { uses: ./.github/workflows/static-analysis.yml }
   multi-compiler:  { uses: ./.github/workflows/nightly-multi-compiler.yml }
+  fips-ready:      { uses: ./.github/workflows/nightly-fips-ready.yml }