Repository navigation
The AArch64 kernel is entered with the MMU off, and refuses an entry with it on - #677
Merged
Merged
Conversation
…with it on Supersedes #676: the same change, cut from main so that history does not carry the firmware images #676's earlier rounds added and removed. ## What it fixes All 16 `virt_*` guests died at the AArch64 kernel's first instruction on CI's firmware, Debian's `AAVMF_CODE.no-secboot.fd` 2026.05-2. The runs are 36863809437 (#671's `guest` check) and 36843762360 (main's nightly at `06788146b`). - Each console ends at the loader's `Loader log: the kernel handoff begins` and then edk2's own `Synchronous Exception at 0x00000000BC33EB20`. That PC is inside the image at `0xbc200000`, and it is the same entered at EL1 and at EL2. - The same tests pass on the dev host's firmware, Homebrew QEMU 11.1.1's own `edk2-stable202408`. ## Root cause **1. The image is not executable under firmware's tables.** The loader allocates the kernel image from the `uefi` crate's pool, which allocates `EfiLoaderData`, and it branched to `_start` with firmware's page tables still live. - edk2's ArmVirtQemu has defaulted `PcdDxeNxMemoryProtectionPolicy` to `0xC000000000007FD5` since `2997ae3873` ("ArmVirtPkg: make EFI_LOADER_DATA non-executable", 2022-08-02). Bit 2 of that value is `EfiLoaderData`, and Debian 2026.05-2 keeps the default. - QEMU builds its own firmware with `pcds = nx.broken.shim.grub`, `0xC000000000007FD1` (`roms/edk2-build.config` at v11.1.1). Each decompressed DXE volume has one `movz x1, #imm; movk x1, #0xc000, lsl #48; tst` site: `0x7fd5` in Debian's 2026.05-2, `0x7fd1` in Homebrew's. - UEFI 2.11 §2.3.6 promises only an identity map of RAM ("The state of other system control register bits is not dictated by this specification"), and maps `EFI_MEMORY_XP` to UXN/PXN at EL1&0 and XN at EL2. Debian's no-secboot build sets `PcdUninstallMemAttrProtocol=TRUE`, so a loader cannot ask for the image to be made executable. **2. A second dependency waited behind the first.** The EL2 entry wrote `HCR_EL2.E2H` clear under firmware's live EL2 tables. - edk2 `47e28a6d44` ("ArmVirtPkg/ArmPlatformLibQemu: Enable early ID map on EL2+VHE") sets `E2H` at EL2, and CI's loader printed `HCR_EL2.E2H 1`. It is first in `edk2-stable202502` (`git tag --contains`); the message of `979551738` says stable202411, which is wrong. - `b8df7d9c8e` (stable202605) makes that early map LPA2 where `PARange` is 52 bits, as on `-cpu max`. Clearing `E2H` makes the CPU read the same `TCR_EL2` in the non-VHE layout, under the PC. ## The firmware ruling this keeps The owner ruled on 2026-09-28 (`dfe61e503`) that a guest's firmware comes from the QEMU installation, inside "Rust and QEMU". Rounds 1 to 4 of this branch committed Debian's edk2 builds on a brief that did not know the ruling. Its last commit restores main's `src/firmware.rs` (the search of QEMU's `firmware/*.json` descriptors that QEMU's interop spec describes) and its callers. It deletes `qemu-firmware/`, its `NOTICE` sections, its two `licenses/` texts, its `COMMITTED_FILES` rows and the per-accelerator table. `nightly.yml` and the host-tools ledger are main's again, so the containers install the same firmware packages as main. So the dev host boots Homebrew's stable202408 and CI boots Debian's 2026.05-2 from its image's snapshot. That is two firmwares, which is how this defect was caught. Homebrew's needs neither the handover nor the EL2 arm below: it leaves `EfiLoaderData` executable (`0x7fd1`), never sets `E2H`, and main's suite passes on it. ## Decisions - **The loader hands over with the MMU off, at the level firmware ran it** (`bootloader/src/arch/aarch64.rs`). That is the state PSCI's `CPU_ON` starts every other CPU in. - It cleans the image, `KernelArgs` and its own last instructions to the point of coherency, and invalidates the instruction cache. It clears `SCTLR_ELx.M` and `.C`, in `sctlr_el2` at EL2, which names EL2's register whatever `E2H` holds. Then it branches to `_start`. - Linux's arm64 `efi_enter_kernel` (v6.1, `efi-entry.S`) does the same, and UEFI allows it: "An application written to this specification may alter the processor execution mode". - Rejected: allocating the image as `EfiLoaderCode`. That still rests on firmware's policy for the type, and keeps the `E2H` flip under a live MMU; m1b measures it. - Rejected: `EFI_MEMORY_ATTRIBUTE_PROTOCOL`, which Debian uninstalls. - **The kernel's EL1 write of `SCTLR_MMU_OFF` goes**, with its constant: neither entry arrives under firmware's tables now. - **The entry refuses `SCTLR_ELx.M` set** at the level it was entered at. Each arm of `apply_declaration` reads its own `SCTLR` and branches to `refused_mmu_on_at_entry` on bit 0, before the EL1 arm's translation registers and before the EL2 arm's `HCR_EL2` write. Main's kernel turned the MMU off at EL1 itself, so a kernel started by a loader that hands over under firmware's tables would have rewritten them under a live walk in silence. The `# Safety` section names what both arms rest on. - **`virt_el2_drop` boots `-cpu cortex-a72`** (`Profile::VirtEl2NoVhe`), which has no FEAT_VHE. - There `HCR_EL2.E2H` is RES0, firmware runs EL2 non-VHE, and the `sctlr_el1` encoding at EL2 names EL1's own register. So only the loader's EL2 arm turns EL2's MMU off. On Debian's 2026.05-2 every `-cpu max` EL2 guest runs with `E2H` set, where that encoding is `SCTLR_EL2`'s, and nothing told the two arms apart. - The test asserts `HCR_EL2.E2H 0,` on the loader's line, the premise that tells them apart. - Without the EL2 arm, EL2's MMU stays on wherever `E2H` is clear at the handover. On CI's lanes that is this guest alone, and Debian's firmware maps the image execute-never at EL2. On the dev host, whose firmware never sets `E2H`, it is every EL2 guest, and the entry's refusal halts each. - No guest test is added. One test's CPU changes, because the behaviour is an exception-level transition under a firmware's translation regime: no type expresses it, the host cannot execute it, and the T14 is x86-64. - `src/arch.rs`'s "The CPU every guest of this architecture gets under `accel`" goes, false now. - **The handover's standing guard is CI's AArch64 lanes on Debian's firmware.** Nothing asserts which firmware CI boots beyond `--ci guest`'s instrument line. - **`issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md` files the third finding as a hazard.** A host QEMU that ships an edk2 from stable202502 to stable202608 hangs every `virt` guest under HVF in `ExitBootServices`. QEMU's HVF reads `ID_AA64PFR0_EL1.GIC` as 0, and those builds then run their GICv2 driver on a GICv3 machine. Homebrew's QEMU 11.1.1 ships stable202408. The file keeps the measured cause, d3's readings and the exit. ## Controls and oracles Measured by the orchestrator at earlier heads of this branch. Where a row names 2026.05-2, its guests booted CI's bytes: the committed `QEMU_EFI.fd` and `QEMU_VARS.fd` were the first 3 MiB and 768 KiB of the package's `AAVMF_CODE.no-secboot.fd` and `AAVMF_VARS.fd`, which hold only zeros after them. | head | run | firmware | result | |---|---|---|---| | `8d74557fa` | n0: `979551738` reverted whole | 2026.05-2 | 0/16: the 14 TCG consoles end in CI's recorded failure, 28 `Synchronous Exception` lines; the 2 HVF guests hang in `ExitBootServices` | | `ef659fc30` | n0 | 2026.05-2 under TCG, 2024.11-5 under HVF | 2/16: the 14 TCG guests red, with 28 `version 2026.05-2` banners and 28 `Synchronous Exception` lines; the 2 HVF guests green | | `8d74557fa` | m1b: n0 with the image as `EfiLoaderCode` | 2026.05-2 | 1/16: `virt_el1_smp` green, so the XN half alone; the 13 EL2 guests red after `HCR_EL2.E2H 1` with no exception, so the `E2H` half; the 2 HVF guests hang in `ExitBootServices` | | `8d74557fa` | h1: the fix | Homebrew's stable202408 | 16/16, in two runs | | `8d74557fa` | d1, d2: the HVF guests | 2026.05-2 | 0/2 each, in `GicV2ExitBootServicesEvent` | | `7af1d3307` | d3: `ID_AA64PFR0_EL1` read by the loader | 2024.11-5 | 4/4; `GIC` 0 under HVF, 1 under TCG | Oracles, each independent of the author: - CI's recorded failure. - UEFI 2.11 §2.3.6 and its attribute table. - The Arm ARM: an `_el1` encoding at EL2 is redirected to EL2's register when `E2H` is set, and `E2H` is RES0 without FEAT_VHE. - Linux v6.1's `efi_enter_kernel`, a differential implementation of the same handoff. - edk2's source and commits, and QEMU v11.1.1's (`hvf.c`, `virt.c`). - Each firmware binary, disassembled. ## Gates, at `b0e2f55d1` | gate | exit | |---|---| | `cargo run -- --ci host`: "Host: 56 step(s), all green"; the licence gate "6 exception(s) stand, and nothing else is refused" | 0 | | `cargo run -- --clippy`: "clippy: 17 invocations clean" | 0 | | `cargo run -- --build-only --arch aarch64` | 0 | | the same with `--kernel-feature boot-actuators --kernel-feature test-actuators` | 0 | | `cargo test --test toyos-build --no-run` | 0 | | C1, C2, D, n0, D with the refusal reverted, the EL1 arm leaving `M` set, and a measurement-only harness patch that boots Debian's 2026.05-2 for TCG `virt` guests, alone, with n0 and with D: each applied to `b0e2f55d1`, built both ways, the harness compiled, reverted, the tree clean. The patches are in a comment on this pull request. | 0 each | ## Unsure - No guest has booted `b0e2f55d1`. The whole suite, C1 and C2 under HVF, n0 and D on Debian's firmware and on Homebrew's, D with the refusal reverted, and the EL1 arm leaving `M` set are the orchestrator's runs. - The cleans of `KernelArgs` (C1) and of the handover's own instructions (C2) are new dependencies, and neither is measured. HVF is the only tier here with real caches. - `cortex-a72` is QEMU's TCG model; no silicon here runs it. - This branch's history keeps the deleted firmware, in `5f9508138`, `7af1d3307` and `ef659fc30`. Its objects take 4,383,389 bytes on disk beyond main's (`git rev-list --objects --disk-usage origin/main..HEAD`), and a merge commit carries them into main's history. ## Size `git diff --shortstat origin/main...HEAD`: 7 files, +176 −57. - Production: the loader +54 −12, the kernel +44 −30, `src/arch.rs` −2. - Tests: the harness +31 −13. - The issue +47. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
Collaborator
Author
|
Evidence at The change is #676's. #676's net diff is
History.
Gates at
No guest run: guests are the orchestrator's. |
Collaborator
Author
|
Guest runs at
C1 and C2 are green under HVF: no tier on this host sees a missing clean, so both cleans rest on the Arm ARM and Linux's |
Collaborator
Author
|
Guest run at |
Collaborator
Author
|
Review, round 2, at Earlier BLOCKERs:
Measured for this review:
BLOCKER
NOTE
REMOVE
Patches. Each one, and each pair in the table, passes e2, a measurement (the EL2 arm leaves --- a/bootloader/src/arch/aarch64.rs
+++ b/bootloader/src/arch/aarch64.rs
@@ -148,7 +148,6 @@
"cmp x9, #(2 << 2)",
"b.ne 3f",
"mrs x9, sctlr_el2",
- "bic x9, x9, #(1 << 0)",
"bic x9, x9, #(1 << 2)",
"msr sctlr_el2, x9",
"b 4f",R1, a mutation: --- a/kernel/src/arch/aarch64/boot.rs
+++ b/kernel/src/arch/aarch64/boot.rs
@@ -131,8 +131,6 @@
"cmp x21, #1",
"b.ne 9f",
// EL1: refused with its MMU on; else the declaration, then translation on.
- "mrs x1, sctlr_el1",
- "tbnz x1, #0, {refuse_mmu}",
"msr mair_el1, x4",
"msr tcr_el1, x2",
"msr ttbr0_el1, x3",R, a mutation: --- a/kernel/src/arch/aarch64/boot.rs
+++ b/kernel/src/arch/aarch64/boot.rs
@@ -132,7 +132,7 @@
"b.ne 9f",
// EL1: refused with its MMU on; else the declaration, then translation on.
"mrs x1, sctlr_el1",
- "tbnz x1, #0, {refuse_mmu}",
+ "tbnz x1, #2, {refuse_mmu}",
"msr mair_el1, x4",
"msr tcr_el1, x2",
"msr ttbr0_el1, x3",
@@ -154,7 +154,7 @@
// the drop.
"2:",
"mrs x1, sctlr_el2",
- "tbnz x1, #0, {refuse_mmu}",
+ "tbnz x1, #2, {refuse_mmu}",
"ldr x1, ={hcr}",
"msr hcr_el2, x1",
"isb",C0, the NOTE's change: --- a/bootloader/src/arch/aarch64.rs
+++ b/bootloader/src/arch/aarch64.rs
@@ -149,13 +149,11 @@
"b.ne 3f",
"mrs x9, sctlr_el2",
"bic x9, x9, #(1 << 0)",
- "bic x9, x9, #(1 << 2)",
"msr sctlr_el2, x9",
"b 4f",
"3:",
"mrs x9, sctlr_el1",
"bic x9, x9, #(1 << 0)",
- "bic x9, x9, #(1 << 2)",
"msr sctlr_el1, x9",
"4:",
"isb",
R or R1 on top of run 1, and R on top of run 2, move that PC. SEND BACK |
Both `bic x9, x9, #(1 << 2)` in `enter_kernel` go, with the prose that named `C`: "and data cache" and the PSCI `CPU_ON` clause in its doc, and `.C` in its SAFETY comment. With `M` clear, `C` governs no access, so clearing it bought nothing. The Arm ARM's shared pseudocode (DDI 0487), as Arm's 2023-03 machine-readable release prints it (ISA_A64_xml_A_profile-2023-03, shared_pseudocode.xml): `AArch64.S1Translate` returns `AArch64.S1DisabledOutput` whenever `AArch64.S1Enabled` is false, which is `SCTLR_ELx.M` clear or, at EL1&0, `HCR_EL2.DC` or `.TGE` set. `S1DisabledOutput` gives a data access Device-nGnRnE, or Normal Inner and Outer Write-Back at EL1&0 where `HCR_EL2.DC` is 1, and an instruction fetch Normal memory cacheable by `SCTLR_ELx.I`; it reads `C` for none of them. `AArch64.S1DCacheEnabled`, the only reader of `C`, has two callers, both on paths with stage 1 enabled: `S1Translate` after its walk, and `S1Walk`. `SCTLR_EL1.C`'s own description (SysReg_xml_A_profile-2023-03, AArch64-sctlr_el1.xml): "When the Effective value of the HCR_EL2.DC bit in the current Security state is 1, the PE ignores SCTLR_EL1.C." Past the handover nothing reads the `C` firmware left: the kernel writes `SCTLR_EL1` whole as its MMU comes on, and never writes `SCTLR_EL2`. The review's REMOVEs go with it: the HVF hazard issue's two sentences on what Homebrew's QEMU and QEMU master ship today, and `virt_el2_drop`'s clause restating `Profile::VirtEl2NoVhe`'s doc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
Collaborator
Author
|
Mutation and measurement patches for A job patch is Q followed by its mutations: Q is measurement only, never landed: every AArch64 guest gets a QMP socket, and a boot timeout appends a Q--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -2144,7 +2144,8 @@
/// The QMP socket `options` asks for, named in `dir`.
fn qmp_socket(dir: &Path, options: &BootOptions) -> Option<PathBuf> {
- options.qmp.then(|| dir.join("qmp.sock"))
+ // MEASUREMENT ONLY: every AArch64 guest gets one, for `where_halted`.
+ (options.qmp || options.profile.arch() == Arch::Aarch64).then(|| dir.join("qmp.sock"))
}
/// Every file one boot owns, so that adding another does not lengthen a
@@ -2225,7 +2226,7 @@
let boot_log = if options.mute {
String::new()
} else {
- wait_for_ready(&mut child, &rx, options, &uart_log)
+ wait_for_ready(&mut child, &rx, options, &uart_log, sockets.qmp.as_deref())
};
QemuInstance {
@@ -2273,14 +2274,72 @@
eprintln!("[serial {seq}] {line}");
}
tx.send(line).is_ok()
+}
+
+/// MEASUREMENT ONLY, never landed: where the boot CPU of a guest that never
+/// reached its marker is, as one `[halt]` line. QMP's `info registers` gives
+/// its PC; the loader's `Kernel arguments:` line says where `KernelArgs` is,
+/// whose `kernel_memory_addr` is the image's physical base and whose
+/// `kernel_elf_addr` is the ELF the loader handed the kernel. Both are read
+/// back with `pmemsave`, and the PC is resolved against that ELF's `.symtab`
+/// by the kernel's own symbolizer, `toyos_elf::SymTab::resolve`.
+fn where_halted(socket: &Path, seen: &str, scratch: &Path) -> String {
+ let read = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+ let mut monitor = QmpMonitor::open(socket);
+ let registers = monitor.human("info registers");
+ let Some(pc) = registers.split("PC=").nth(1).and_then(|s| u64::from_str_radix(s.get(..16)?, 16).ok())
+ else {
+ return format!("[halt] no PC in `info registers`:\n{registers}");
+ };
+ let pstate = registers.split("PSTATE=").nth(1).and_then(|s| s.lines().next()).unwrap_or("?");
+ let at = resolve_in_guest(&mut monitor, pc, seen, scratch).unwrap_or_else(|why| format!("unresolved: {why}"));
+ format!("[halt] PC {pc:#x} is {at}; PSTATE={pstate}\n[halt] info registers:\n{registers}")
+ }));
+ match read {
+ Ok(line) => format!("\n{line}"),
+ Err(_) => "\n[halt] the QMP reading itself panicked; its message is above".to_string(),
+ }
+}
+
+fn resolve_in_guest(monitor: &mut QmpMonitor, pc: u64, seen: &str, scratch: &Path) -> Result<String, String> {
+ use toyos_abi::boot::KernelArgs;
+ let args = seen
+ .lines()
+ .find_map(|l| u64::from_str_radix(l.split("Kernel arguments: 0x").nth(1)?.split('+').next()?, 16).ok())
+ .ok_or("the loader never said where KernelArgs is")?;
+ let raw = pmem(monitor, args, std::mem::size_of::<KernelArgs>(), &scratch.with_extension("halt-args"))?;
+ let word = |at: usize| u64::from_le_bytes(raw[at..at + 8].try_into().expect("eight bytes"));
+ let base = word(std::mem::offset_of!(KernelArgs, kernel_memory_addr));
+ let (elf_at, elf_len) =
+ (word(std::mem::offset_of!(KernelArgs, kernel_elf_addr)), word(std::mem::offset_of!(KernelArgs, kernel_elf_size)));
+ let elf = pmem(monitor, elf_at, elf_len as usize, &scratch.with_extension("halt-elf"))?;
+ let (syms, strs) = toyos_symbols::locate(&elf).ok_or("the handed ELF has no readable `.symtab`")?;
+ let table = toyos_elf::SymTab::new(syms, strs);
+ Ok(match pc.checked_sub(base).and_then(|offset| table.resolve(offset)) {
+ Some((name, within)) => format!("{name}+{within:#x}, in the kernel image at {base:#x}"),
+ None => format!("in no kernel function; the kernel image is at {base:#x}, its ELF at {elf_at:#x}+{elf_len:#x}"),
+ })
}
+/// `bytes` of guest physical memory at `phys`, through `pmemsave` into `out`.
+fn pmem(monitor: &mut QmpMonitor, phys: u64, bytes: usize, out: &Path) -> Result<Vec<u8>, String> {
+ let _ = fs::remove_file(out);
+ let command = format!("pmemsave {phys:#x} {bytes} \\\"{}\\\"", out.display());
+ let said = monitor.human(&command);
+ let read = fs::read(out).map_err(|e| format!("{command:?} wrote no file ({e}); the monitor said {said:?}"))?;
+ if read.len() != bytes {
+ return Err(format!("pmemsave {phys:#x} wrote {} bytes and not {bytes}; the monitor said {said:?}", read.len()));
+ }
+ Ok(read)
+}
+
/// Returns every line seen on the way to the marker — see [`QemuInstance::boot_log`].
fn wait_for_ready(
child: &mut Child,
rx: &Receiver<String>,
options: &BootOptions,
uart_log: &Path,
+ qmp: Option<&Path>,
) -> String {
let no_timeout = options.debug_wait;
let ready = options.ready_marker;
@@ -2313,13 +2372,14 @@
let mut seen = String::new();
loop {
if !no_timeout && start.elapsed() > boot_timeout {
+ let halted = qmp.map(|socket| where_halted(socket, &seen, uart_log)).unwrap_or_default();
let _ = child.kill();
// With what it did say. A timeout that discards the console is the
// one failure in this harness that arrives with no evidence at all,
// and "the guest printed nothing" and "the guest printed sixty
// lines and then stopped" are different machines.
panic!(
- "[qemu] Boot timed out waiting for {ready}; the console carried:\n{}",
+ "[qemu] Boot timed out waiting for {ready}; the console carried:\n{}{halted}",
if seen.is_empty() { "nothing at all".to_string() } else { seen.clone() }
);
}R--- a/kernel/src/arch/aarch64/boot.rs
+++ b/kernel/src/arch/aarch64/boot.rs
@@ -132,7 +132,7 @@
"b.ne 9f",
// EL1: refused with its MMU on; else the declaration, then translation on.
"mrs x1, sctlr_el1",
- "tbnz x1, #0, {refuse_mmu}",
+ "tbnz x1, #2, {refuse_mmu}",
"msr mair_el1, x4",
"msr tcr_el1, x2",
"msr ttbr0_el1, x3",
@@ -154,7 +154,7 @@
// the drop.
"2:",
"mrs x1, sctlr_el2",
- "tbnz x1, #0, {refuse_mmu}",
+ "tbnz x1, #2, {refuse_mmu}",
"ldr x1, ={hcr}",
"msr hcr_el2, x1",
"isb",R1--- a/kernel/src/arch/aarch64/boot.rs
+++ b/kernel/src/arch/aarch64/boot.rs
@@ -131,8 +131,6 @@
"cmp x21, #1",
"b.ne 9f",
// EL1: refused with its MMU on; else the declaration, then translation on.
- "mrs x1, sctlr_el1",
- "tbnz x1, #0, {refuse_mmu}",
"msr mair_el1, x4",
"msr tcr_el1, x2",
"msr ttbr0_el1, x3",E1--- a/bootloader/src/arch/aarch64.rs
+++ b/bootloader/src/arch/aarch64.rs
@@ -152,7 +152,6 @@
"b 4f",
"3:",
"mrs x9, sctlr_el1",
- "bic x9, x9, #(1 << 0)",
"msr sctlr_el1, x9",
"4:",
"isb",e2--- a/bootloader/src/arch/aarch64.rs
+++ b/bootloader/src/arch/aarch64.rs
@@ -147,7 +147,6 @@
"cmp x9, #(2 << 2)",
"b.ne 3f",
"mrs x9, sctlr_el2",
- "bic x9, x9, #(1 << 0)",
"msr sctlr_el2, x9",
"b 4f",
"3:", |
Collaborator
Author
|
Guest runs at
C0 gives the refusal's bit a standing red (R reds all 16). E1 against E1-R1 shows the EL1 check is what halts those three guests: without it, the kernel entered at EL1 with |
Collaborator
Author
|
Review, round 6, at Earlier BLOCKER:
Measured for this review:
BLOCKER
NOTE
REMOVE
LAND AFTER NAMED CHANGES |
Japabu
added a commit
that referenced
this pull request
Oct 1, 2026
#677 (76d0d93) touches no file this branch does: the AArch64 loader hands the kernel over with the MMU off, and the kernel refuses an entry with it on. It is the fix `issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md` waits on; that issue's exit is CI's `guest` check, which this merge is the first of this branch to run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
Japabu
added a commit
that referenced
this pull request
Oct 2, 2026
…ernel issues CI met Filed, each with its owner, evidence and exit: - a runner image that moves cc, c++ or CMake moves every LLVM key, so every pull request and merge group builds the cold path until main saves the new keys; - a toolchain job that ends inside `--ci bootstrap` saves no layer it built: run 36913380100 built all four, reded in that step and saved none, and run 36934214557 built the LLVM again. Closed on job 110649069031 (run 36934214557, `guest / suite` under KVM on the merge of f1ccb0b into 76d0d93: "test result: ok. 21 passed, 21 total"), which meets both exits as written: - every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware: the sixteen `virt_*` tests are green in CI's `guest` check, since #677 enters the AArch64 kernel with the MMU off; - the-nested-nmi-report-interleaves-with-another-cpus-console-line: `nested_nmi_is_loud` is green there, since #675 writes the report under the console's registers, which `nested_nmi`'s own comment says. Neither was cited outside the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it fixes
All 16
virt_*guests died at the AArch64 kernel's first instruction on CI's firmware, Debian'sAAVMF_CODE.no-secboot.fd2026.05-2. The runs are 36863809437 (#671'sguestcheck) and 36843762360 (main's nightly at06788146b).Loader log: the kernel handoff beginsand then edk2's ownSynchronous Exception at 0x00000000BC33EB20. That PC is inside the image at0xbc200000, and it is the same entered at EL1 and at EL2.edk2-stable202408.Root cause
1. The image is not executable under firmware's tables. The loader allocates the kernel image from the
ueficrate's pool, which allocatesEfiLoaderData, and it branched to_startwith firmware's page tables still live.PcdDxeNxMemoryProtectionPolicyto0xC000000000007FD5since2997ae3873("ArmVirtPkg: make EFI_LOADER_DATA non-executable", 2022-08-02). Bit 2 of that value isEfiLoaderData, and Debian 2026.05-2 keeps the default.pcds = nx.broken.shim.grub,0xC000000000007FD1(roms/edk2-build.configat v11.1.1). Each decompressed DXE volume has onemovz x1, #imm; movk x1, #0xc000, lsl #48; tstsite:0x7fd5in Debian's 2026.05-2,0x7fd1in Homebrew's.EFI_MEMORY_XPto UXN/PXN at EL1&0 and XN at EL2. Debian's no-secboot build setsPcdUninstallMemAttrProtocol=TRUE, so a loader cannot ask for the image to be made executable.2. A second dependency waited behind the first. The EL2 entry wrote
HCR_EL2.E2Hclear under firmware's live EL2 tables.47e28a6d44("ArmVirtPkg/ArmPlatformLibQemu: Enable early ID map on EL2+VHE") setsE2Hat EL2, and CI's loader printedHCR_EL2.E2H 1. It is first inedk2-stable202502(git tag --contains).b8df7d9c8e(stable202605) makes that early map LPA2 wherePARangeis 52 bits, as on-cpu max. ClearingE2Hmakes the CPU read the sameTCR_EL2in the non-VHE layout, under the PC.The firmware ruling this keeps
The owner ruled on 2026-09-28 (
dfe61e503) that a guest's firmware comes from the QEMU installation, inside "Rust and QEMU".So the dev host boots Homebrew's stable202408 and CI boots Debian's 2026.05-2 from its image's snapshot. That is two firmwares, which is how this defect was caught. Homebrew's needs neither the handover nor the EL2 arm below: it leaves
EfiLoaderDataexecutable (0x7fd1), never setsE2H, and main's suite passes on it.Decisions
bootloader/src/arch/aarch64.rs).KernelArgsand its own last instructions to the point of coherency, and invalidates the instruction cache. It clearsSCTLR_ELx.M, insctlr_el2at EL2, which names EL2's register whateverE2Hholds. Then it branches to_start.efi_enter_kernel(v6.1,efi-entry.S) hands over the same way, clearingCbesides, and UEFI allows it: "An application written to this specification may alter the processor execution mode".EfiLoaderCode. That still rests on firmware's policy for the type, and keeps theE2Hflip under a live MMU; m1b measures it.EFI_MEMORY_ATTRIBUTE_PROTOCOL, which Debian uninstalls.SCTLR_ELx.Cstays as firmware left it. WithMclear it governs no access. In the Arm ARM's pseudocode,AArch64.S1TranslatereturnsAArch64.S1DisabledOutputwhenever stage 1 is disabled, and that gives a data access Device-nGnRnE, or Normal Write-Back at EL1&0 whereHCR_EL2.DCis 1, without readingC;AArch64.S1DCacheEnabled, the one reader ofC, is called only with stage 1 enabled.SCTLR_EL1.C: "When the Effective value of the HCR_EL2.DC bit in the current Security state is 1, the PE ignores SCTLR_EL1.C." The kernel writesSCTLR_EL1whole as its MMU comes on and never writesSCTLR_EL2.SCTLR_MMU_OFFgoes, with its constant: neither entry arrives under firmware's tables now.SCTLR_ELx.Mset at the level it was entered at. Each arm ofapply_declarationreads its ownSCTLRand branches torefused_mmu_on_at_entryon bit 0, before the EL1 arm's translation registers and before the EL2 arm'sHCR_EL2write. The# Safetysection names what both arms rest on.serial::init) and draws its panel only once the declaration it refuses has turned on the map both are reached through. Its symbol is what names it, so the controls below read the halted PC over QMP.virt_el2_dropboots-cpu cortex-a72(Profile::VirtEl2NoVhe), which has no FEAT_VHE.HCR_EL2.E2His RES0, firmware runs EL2 non-VHE, and thesctlr_el1encoding at EL2 names EL1's own register. So only the loader's EL2 arm turns EL2's MMU off. On Debian's 2026.05-2 every-cpu maxEL2 guest runs withE2Hset, where that encoding isSCTLR_EL2's, and nothing told the two arms apart.HCR_EL2.E2H 0,on the loader's line, the premise that tells them apart.E2His clear at the handover. On CI's lanes that is this guest alone, and Debian's firmware maps the image execute-never at EL2. On the dev host, whose firmware never setsE2H, it is every EL2 guest, and the entry's refusal halts each.src/arch.rs's "The CPU every guest of this architecture gets underaccel" goes, false now.--ci guest's instrument line.issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.mdfiles the third finding as a hazard. A host QEMU that ships an edk2 from stable202502 to stable202608 hangs everyvirtguest under HVF inExitBootServices. QEMU's HVF readsID_AA64PFR0_EL1.GICas 0, and those builds then run their GICv2 driver on a GICv3 machine. Homebrew's QEMU 11.1.1 ships stable202408. The file keeps the measured cause, d3's readings and the exit.Controls and oracles
The orchestrator's, on the dev host (M4 Pro, Homebrew QEMU 11.1.1 with its edk2-stable202408):
virt_early_*under HVF, the othervirt_*and x86-64 under TCG.Measured on
b0e2f55d1(#676), whose tree is this branch's first commit's change:75aa4afec^{tree}isgit merge-tree --write-tree origin/main b0e2f55d1.f182b6cfaadds C0 to it. Patches: #676, issuecomment-5937056807.debis measurement only: every TCGvirtguest boots Debian's 2026.05-2AAVMF_CODE.no-secboot.fdandAAVMF_VARS.fd, the bytes CI's AArch64 lanes boot, and each boot log is asserted to carry that firmware's banner.KernelArgsnot cleanedvirt_early_(HVF)virt_early_(HVF)virt_bootloader/andkernel/at main'svirt_virt_virt_el2_drop(cortex-a72) redbootloader/andkernel/at main'svirt_virt_virt_Msetvirt_virt_el1_smp,virt_early_panic,virt_early_faultredvirt_el2_drop.efi_enter_kernel, not on a measured red.At
75aa4afec: the whole suite, 21/21 (issuecomment-5937436034).m1b, on
8d74557fa(#676) with Debian's 2026.05-2: main's handover with the image allocated asEfiLoaderCodeand the kernel's reservation accepting it, 1/16.virt_el1_smpgreen, so the XN half alone; the 13 EL2 guests red afterHCR_EL2.E2H 1with no exception, so theE2Hhalf; the 2 HVF guests hang inExitBootServices.This round's runs, at
f182b6cfa, the orchestrator's (issuecomment-5938455090)Every patch here carries Q, a measurement-only harness change: each AArch64 guest gets a QMP socket, and a boot timeout appends a
[halt]line, the boot CPU's PC from QMP'sinfo registers, resolved bytoyos_elf::SymTab::resolveagainst the kernel ELF the loader handed the guest (KernelArgs, found by the loader'sKernel arguments:line, and itskernel_elf_addr, each read back withpmemsave). R and R1 are the review's patches; E1 and e2 delete thebic #(1 << 0)of the loader's EL1 and EL2 arm. Patches: a comment on this pull request.a64handover-r6-wholea64handover-r6-Rvirt_[halt]inrefused_mmu_on_at_entry[halt]atrefused_mmu_on_at_entry+0x4, at EL1h and EL2ha64handover-r6-E1(run 1)virt_evirt_el2_dropgreen;virt_el1_smp,virt_early_panic,virt_early_faultred, each[halt]inrefused_mmu_on_at_entryat EL1[halt]atrefused_mmu_on_at_entry+0x4at EL1ha64handover-r6-e2(run 2)virt_el2_drop[halt]inrefused_mmu_on_at_entryat EL2[halt]atrefused_mmu_on_at_entry+0x4at EL2ha64handover-r6-E1-R1virt_e[halt]inrefused_mmu_on_at_entry; each EL1 guest green, or red with its[halt]elsewhereOracles, each independent of the author:
_el1encoding at EL2 is redirected to EL2's register whenE2His set, andE2His RES0 without FEAT_VHE. For C0, its shared pseudocode (AArch64.S1Translate,AArch64.S1DisabledOutput,AArch64.S1DCacheEnabled) andSCTLR_EL1.C,SCTLR_EL2.CandHCR_EL2.DC, read in Arm's 2023-03 machine-readable release (ISA_A64_xml_A_profile-2023-03, sha256 4be65585693d1bf1f1765bcc1a2493ce5df99718cc33d6e98fc3e83f76019bfd;SysReg_xml_A_profile-2023-03, sha256 a136e9eef95aed87b6f9b50ec0c328c9f5966e04d61f66b759cc098a3c0e44a4).efi_enter_kernel, a differential implementation of the same handoff.hvf.c,virt.c).Gates
At
f182b6cfa:cargo run -- --ci host: "[ci] Host: 59 step(s), all green"; the licence gate "6 exception(s) stand, and nothing else is refused"cargo run -- --clippy: "clippy: 17 invocations clean"cargo run -- --build-only --arch aarch64: "Build finished."--kernel-feature boot-actuators --kernel-feature test-actuators: "Build finished."cargo test --test toyos-build --no-runf182b6cfa, built both ways, the harness compiled, reverted, the tree cleanAt
75aa4afec(issuecomment-5937375906):cargo run -- --ci host: "[ci] Host: 59 step(s), all green"; the licence gate "6 exception(s) stand, and nothing else is refused"cargo run -- --clippy: "clippy: 17 invocations clean"cargo run -- --build-only --arch aarch64: "Build finished."cargo test --test toyos-build --no-runUnsure
cortex-a72is QEMU's TCG model; no silicon here runs it.Size
git diff --shortstat origin/main...HEAD: 7 files, +167 −52.src/arch.rs−2.🤖 Generated with Claude Code
https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L