ci: Add experimental RHEL 10 runner coverage by cgwalters-bot · Pull Request #2470 · bootc-dev/bootc

cgwalters-bot · 2026-09-17T02:17:24Z

Motivation

Move bootc CI workloads to the stock GitHub-hosted RHEL 10 partner runner without maintaining a parallel RHEL workflow, while retaining one Ubuntu 26.04 compatibility lane.

Design

This draft depends on bootc-dev/actions#60 and temporarily uses cgwalters-bot/actions/bootc-host-setup@bootc-host-setup-rhel10. That action owns distro-specific dependency installation and the RHEL libvirt workaround. ci.yml remains the sole workflow: the existing workload matrices move to RHEL 10, while one Ubuntu 26.04 job builds the CentOS 10 package and runs the readonly TMT plan with it.

The xtask VM launch paths also remove Cargo's inherited LD_LIBRARY_PATH before invoking bcvk libvirt run, so host QEMU does not accidentally load build-tool libraries.

RHEL libvirt investigation

The remaining RHEL failure was caused by libvirt's per-domain sVirt labeling conflicting with the readonly virtiofs export of rootless Podman storage used by bcvk --bind-storage-ro. The host setup action now writes the unprivileged session config at ${XDG_CONFIG_HOME:-$HOME/.config}/libvirt/qemu.conf with security_driver = "none" on RHEL when libvirt is requested, and validates its exact contents. Host SELinux remains Enforcing; this only disables per-domain sVirt confinement for the ephemeral unprivileged libvirt session.

Controlled bootc run 35322108262 passed the isolated RHEL readonly workload with this configuration. Actions PR #60's technical checks, including its RHEL 10 host setup and libvirt jobs, also pass.

Validation

cargo fmt --check
cargo check -p xtask
cargo test -p xtask (13 passed)
cargo xtask update-generated direct --check
git diff --check
actionlint with the repository's pre-existing disabled-job diagnostic ignored
Independent review found no correctness or workflow-gating defects
Full validation run 35342593795: package jobs, Ubuntu compatibility, upgrades, baseconfigs, container export, and all four sealed UKI integration jobs passed; four unsealed GRUB jobs failed in the pre-existing plan-44-shadow-fixup (two missing-group assertions and two guest reboot timeouts) after integration was moved to the reviewer-requested 4c runner.

AI disclosure

Assisted-by: AI

The workflow and xtask changes were AI-assisted and validated with the commands above. Human review is required; commits intentionally have no Signed-off-by trailers.

cgwalters · 2026-09-17T10:57:54Z

+        with:
+          persist-credentials: false
+
+      - name: Install pinned just


I think we can get this from EPEL

cgwalters · 2026-09-17T10:58:18Z

+        run: sudo dnf install -y gcc
+
+      - name: Install stable Rust
+        uses: dtolnay/rust-toolchain@stable


Use packaged Rust instead.

cgwalters · 2026-09-17T11:00:13Z

+          # Resolve each package against the enabled RHEL repositories rather
+          # than assuming Ubuntu package names.  The alternatives are needed


This whole thing is gross. Let's move it out into a clean separate ./ci/install-deps.sh. Even better, also use the declarative "packages.txt" pattern from e.g. https://github.com/coreos/coreos-assembler/blob/main/src/deps.txt (see in the source where we pipe that to dnf install).

cgwalters · 2026-09-17T11:02:30Z

+          # bcvk requests ordinary insecure UEFI; disable only that descriptor
+          # on this disposable host so libvirt does not select SEV firmware for
+          # a plain KVM guest.
+          for firmware in /usr/share/qemu/firmware/*amdsev*.json; do


This one should be a clear ./ci/workarounds.sh or something

cgwalters · 2026-09-17T11:02:51Z

+        timeout-minutes: 10
+        run: |
+          set -euo pipefail
+          printf '%s\n' '--- KVM, CPU, permissions, firmware, and virtiofsd ---'


./ci/test-environment.sh

cgwalters · 2026-09-17T11:05:01Z

+      - name: Collect host diagnostics
+        run: |
+          set +e
+          timeout 30s bash -c 'dnf repolist; dnf list --available podman git-core curl tar gzip' || true


OK this is too much ugly bash code.

Our real goal is to cut over the main ci.yml to have runs-on: rhel10, not two separate CI jobs.

Let's migrate https://github.com/bootc-dev/actions/tree/main/bootc-ubuntu-setup to be https://github.com/bootc-dev/actions/tree/main/bootc-host-setup which handles both rhel and ubuntu hosts.

cgwalters · 2026-09-17T11:05:26Z

+          timeout --foreground --signal=TERM --kill-after=60s 100m \
+            env BOOTC_jobs=8 just test-tmt readonly
+
+      - name: Collect VM and TMT diagnostics


I don't think we need this anymore

cgwalters · 2026-09-17T11:05:52Z

+          # bcvk is distributed as a release binary; it is not an Ubuntu/RHEL
+          # package.  Keep the upstream tool version used by the existing VM
+          # setup, while leaving action references mutable as this project does.
+          export BCVK_VERSION=0.19.0


Needs to be wired to renovate (let's add an AGENTS.md in this dir that notes this, no forever-pinned deps).

Now in this case, we SHOULD be getting it from EPEL too but this reveals we aren't updating EPEL...let's make that a clear TODO and continue to fetch the GH binary directly.

cgwalters · 2026-09-17T14:35:43Z

+        // not let libvirt propagate it to the host QEMU process: QEMU may
+        // resolve host libraries from target/debug and exit before its monitor
+        // is available.
+        .env_remove("LD_LIBRARY_PATH")


cargo run injects this search path for Rust test binaries.

Ugh. I'm confused though...why didn't this break on Ubuntu? And what shared libraries are involved here?

cgwalters · 2026-09-17T17:24:25Z

      - name: Run TMT readonly tests
        run: |
-          cargo xtask run-tmt \
+          ci/workarounds.sh selinux-permissive -- cargo xtask run-tmt \


I dunno, this is such a big portion of the test suite I'm not totally sure it's better than just globally setting permissive.

But that said I really want to root cause the denial here, include referenced avc logs and search selinux policy

cgwalters-bot · 2026-09-18T00:21:32Z

CI update: run 35285960807 reached terminal failure after docs, validate, cargo-deny, and all six package jobs passed. All downstream upgrade/baseconfig/integration jobs started and installed tmt successfully, but VM creation consistently failed at libvirt with QEMU unexpectedly closed the monitor / process exited while connecting to monitor. The final rerun has no LD_LIBRARY_PATH in the failed job environment, so this is not evidence for restoring the removed LD workaround. The available VM log artifacts contain only the empty pre-launch journal files, so the QEMU exit reason is not yet exposed.

Earlier migration defects were fixed and rerun: host setup now precedes checkout (the stock image lacked git), actions PR #60 adds libvirt-devel for tmt, and install-tmt exposes its user binary path. Actions PR #60 checks are green. This remains an unresolved RHEL libvirt/QEMU runner blocker.

cgwalters · 2026-09-18T11:14:50Z

  # Run basic validation checks (linting, formatting, etc)
  validate:
-    runs-on: ubuntu-26.04
+    runs-on: rhel10-x86_64-16c-64g


Actually a super important thing to notice here is not just that we're going to RHEL as the default runner, but we're doing a much larger runner by default, which should hopefully make at least the Rust builds faster.

That said, we are also definitely going to need to parallelize our tmt tests among others, and also consider the impact of the testing matrix in general vs larger runners.

Hmm, perhaps until our tmt VM tests aren't parallelized we should actually change the integration tests to use the 4c-16g instance? Yeah we should do that...

Done. The test-integration matrix now uses rhel10-x86_64-4c-16g; package and other workloads remain on the larger runner where their parallel builds can use it. I also added the new label to the actionlint configuration.

Keep the final SSH client diagnostic from the readiness retry loop so transport, authentication, and execution failures are actionable without flooding CI logs. Assisted-by: AI

Move the existing CI workloads to the stock RHEL 10 partner runner rather than maintaining a duplicate experimental workflow. The temporary shared host action keeps Ubuntu-specific setup out of this repository and provides the RHEL-packaged Rust and Cargo toolchain. Run host setup before checkout because the stock RHEL image does not provide git. Assisted-by: AI

Build tooling augments LD_LIBRARY_PATH while running xtask. Libvirt can propagate that environment into host QEMU processes, causing incompatible libraries to make QEMU exit before its monitor is available on RHEL 10. Keep inherited library search paths out of every bcvk VM launch. Assisted-by: AI

cgwalters-bot · 2026-09-18T15:11:21Z

Validation run 35342593795: package jobs, Ubuntu compatibility, upgrades, baseconfigs, container export, and all four sealed UKI integration jobs passed. Four unsealed GRUB jobs failed in the pre-existing plan-44-shadow-fixup (two missing-group assertions and two guest reboot timeouts) after integration moved to the reviewer-requested 4c runner. The original sVirt/virtiofs and missing virt-fw-vars failures are resolved; shadow-fixup needs separate follow-up.

bootc-bot Bot requested a review from cgwalters September 17, 2026 02:17

cgwalters requested changes Sep 17, 2026

View reviewed changes

cgwalters-bot force-pushed the rhel10-ci-experiment branch from 7e41b8f to fcd1698 Compare September 17, 2026 12:54

cgwalters requested changes Sep 17, 2026

View reviewed changes

cgwalters-bot force-pushed the rhel10-ci-experiment branch 5 times, most recently from ac8ced1 to e2b4355 Compare September 17, 2026 23:14

cgwalters-bot force-pushed the rhel10-ci-experiment branch 10 times, most recently from 0e3cf8e to 70f0a7f Compare September 18, 2026 11:14

cgwalters reviewed Sep 18, 2026

View reviewed changes

cgwalters added 3 commits September 18, 2026 08:01

tmt: Report SSH readiness failures

a3dc4f1

Keep the final SSH client diagnostic from the readiness retry loop so transport, authentication, and execution failures are actionable without flooding CI logs. Assisted-by: AI

cgwalters-bot force-pushed the rhel10-ci-experiment branch from 70f0a7f to ae1840a Compare September 18, 2026 12:02

		# Resolve each package against the enabled RHEL repositories rather
		# than assuming Ubuntu package names. The alternatives are needed

Conversation

cgwalters-bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Design

RHEL libvirt investigation

Validation

AI disclosure

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters-bot commented Sep 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters-bot commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cgwalters-bot commented Sep 17, 2026 •

edited

Loading