From 75aa4afec9e919314c8468a890ea341743af5bc4 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 19:49:44 +0200 Subject: [PATCH 1/2] The AArch64 kernel is entered with the MMU off, and refuses an entry with it on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Supersedes #676: the same change, cut from main so that history does not carry the firmware images #676's earlier rounds added and removed. ## What it fixes All 16 `virt_*` guests died at the AArch64 kernel's first instruction on CI's firmware, Debian's `AAVMF_CODE.no-secboot.fd` 2026.05-2. The runs are 36863809437 (#671's `guest` check) and 36843762360 (main's nightly at `06788146b`). - Each console ends at the loader's `Loader log: the kernel handoff begins` and then edk2's own `Synchronous Exception at 0x00000000BC33EB20`. That PC is inside the image at `0xbc200000`, and it is the same entered at EL1 and at EL2. - The same tests pass on the dev host's firmware, Homebrew QEMU 11.1.1's own `edk2-stable202408`. ## Root cause **1. The image is not executable under firmware's tables.** The loader allocates the kernel image from the `uefi` crate's pool, which allocates `EfiLoaderData`, and it branched to `_start` with firmware's page tables still live. - edk2's ArmVirtQemu has defaulted `PcdDxeNxMemoryProtectionPolicy` to `0xC000000000007FD5` since `2997ae3873` ("ArmVirtPkg: make EFI_LOADER_DATA non-executable", 2022-08-02). Bit 2 of that value is `EfiLoaderData`, and Debian 2026.05-2 keeps the default. - QEMU builds its own firmware with `pcds = nx.broken.shim.grub`, `0xC000000000007FD1` (`roms/edk2-build.config` at v11.1.1). Each decompressed DXE volume has one `movz x1, #imm; movk x1, #0xc000, lsl #48; tst` site: `0x7fd5` in Debian's 2026.05-2, `0x7fd1` in Homebrew's. - UEFI 2.11 §2.3.6 promises only an identity map of RAM ("The state of other system control register bits is not dictated by this specification"), and maps `EFI_MEMORY_XP` to UXN/PXN at EL1&0 and XN at EL2. Debian's no-secboot build sets `PcdUninstallMemAttrProtocol=TRUE`, so a loader cannot ask for the image to be made executable. **2. A second dependency waited behind the first.** The EL2 entry wrote `HCR_EL2.E2H` clear under firmware's live EL2 tables. - edk2 `47e28a6d44` ("ArmVirtPkg/ArmPlatformLibQemu: Enable early ID map on EL2+VHE") sets `E2H` at EL2, and CI's loader printed `HCR_EL2.E2H 1`. It is first in `edk2-stable202502` (`git tag --contains`); the message of `979551738` says stable202411, which is wrong. - `b8df7d9c8e` (stable202605) makes that early map LPA2 where `PARange` is 52 bits, as on `-cpu max`. Clearing `E2H` makes the CPU read the same `TCR_EL2` in the non-VHE layout, under the PC. ## The firmware ruling this keeps The owner ruled on 2026-09-28 (`dfe61e503`) that a guest's firmware comes from the QEMU installation, inside "Rust and QEMU". Rounds 1 to 4 of this branch committed Debian's edk2 builds on a brief that did not know the ruling. Its last commit restores main's `src/firmware.rs` (the search of QEMU's `firmware/*.json` descriptors that QEMU's interop spec describes) and its callers. It deletes `qemu-firmware/`, its `NOTICE` sections, its two `licenses/` texts, its `COMMITTED_FILES` rows and the per-accelerator table. `nightly.yml` and the host-tools ledger are main's again, so the containers install the same firmware packages as main. So the dev host boots Homebrew's stable202408 and CI boots Debian's 2026.05-2 from its image's snapshot. That is two firmwares, which is how this defect was caught. Homebrew's needs neither the handover nor the EL2 arm below: it leaves `EfiLoaderData` executable (`0x7fd1`), never sets `E2H`, and main's suite passes on it. ## Decisions - **The loader hands over with the MMU off, at the level firmware ran it** (`bootloader/src/arch/aarch64.rs`). That is the state PSCI's `CPU_ON` starts every other CPU in. - It cleans the image, `KernelArgs` and its own last instructions to the point of coherency, and invalidates the instruction cache. It clears `SCTLR_ELx.M` and `.C`, in `sctlr_el2` at EL2, which names EL2's register whatever `E2H` holds. Then it branches to `_start`. - Linux's arm64 `efi_enter_kernel` (v6.1, `efi-entry.S`) does the same, and UEFI allows it: "An application written to this specification may alter the processor execution mode". - Rejected: allocating the image as `EfiLoaderCode`. That still rests on firmware's policy for the type, and keeps the `E2H` flip under a live MMU; m1b measures it. - Rejected: `EFI_MEMORY_ATTRIBUTE_PROTOCOL`, which Debian uninstalls. - **The kernel's EL1 write of `SCTLR_MMU_OFF` goes**, with its constant: neither entry arrives under firmware's tables now. - **The entry refuses `SCTLR_ELx.M` set** at the level it was entered at. Each arm of `apply_declaration` reads its own `SCTLR` and branches to `refused_mmu_on_at_entry` on bit 0, before the EL1 arm's translation registers and before the EL2 arm's `HCR_EL2` write. Main's kernel turned the MMU off at EL1 itself, so a kernel started by a loader that hands over under firmware's tables would have rewritten them under a live walk in silence. The `# Safety` section names what both arms rest on. - **`virt_el2_drop` boots `-cpu cortex-a72`** (`Profile::VirtEl2NoVhe`), which has no FEAT_VHE. - There `HCR_EL2.E2H` is RES0, firmware runs EL2 non-VHE, and the `sctlr_el1` encoding at EL2 names EL1's own register. So only the loader's EL2 arm turns EL2's MMU off. On Debian's 2026.05-2 every `-cpu max` EL2 guest runs with `E2H` set, where that encoding is `SCTLR_EL2`'s, and nothing told the two arms apart. - The test asserts `HCR_EL2.E2H 0,` on the loader's line, the premise that tells them apart. - Without the EL2 arm, EL2's MMU stays on wherever `E2H` is clear at the handover. On CI's lanes that is this guest alone, and Debian's firmware maps the image execute-never at EL2. On the dev host, whose firmware never sets `E2H`, it is every EL2 guest, and the entry's refusal halts each. - No guest test is added. One test's CPU changes, because the behaviour is an exception-level transition under a firmware's translation regime: no type expresses it, the host cannot execute it, and the T14 is x86-64. - `src/arch.rs`'s "The CPU every guest of this architecture gets under `accel`" goes, false now. - **The handover's standing guard is CI's AArch64 lanes on Debian's firmware.** Nothing asserts which firmware CI boots beyond `--ci guest`'s instrument line. - **`issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md` files the third finding as a hazard.** A host QEMU that ships an edk2 from stable202502 to stable202608 hangs every `virt` guest under HVF in `ExitBootServices`. QEMU's HVF reads `ID_AA64PFR0_EL1.GIC` as 0, and those builds then run their GICv2 driver on a GICv3 machine. Homebrew's QEMU 11.1.1 ships stable202408. The file keeps the measured cause, d3's readings and the exit. ## Controls and oracles Measured by the orchestrator at earlier heads of this branch. Where a row names 2026.05-2, its guests booted CI's bytes: the committed `QEMU_EFI.fd` and `QEMU_VARS.fd` were the first 3 MiB and 768 KiB of the package's `AAVMF_CODE.no-secboot.fd` and `AAVMF_VARS.fd`, which hold only zeros after them. | head | run | firmware | result | |---|---|---|---| | `8d74557fa` | n0: `979551738` reverted whole | 2026.05-2 | 0/16: the 14 TCG consoles end in CI's recorded failure, 28 `Synchronous Exception` lines; the 2 HVF guests hang in `ExitBootServices` | | `ef659fc30` | n0 | 2026.05-2 under TCG, 2024.11-5 under HVF | 2/16: the 14 TCG guests red, with 28 `version 2026.05-2` banners and 28 `Synchronous Exception` lines; the 2 HVF guests green | | `8d74557fa` | m1b: n0 with the image as `EfiLoaderCode` | 2026.05-2 | 1/16: `virt_el1_smp` green, so the XN half alone; the 13 EL2 guests red after `HCR_EL2.E2H 1` with no exception, so the `E2H` half; the 2 HVF guests hang in `ExitBootServices` | | `8d74557fa` | h1: the fix | Homebrew's stable202408 | 16/16, in two runs | | `8d74557fa` | d1, d2: the HVF guests | 2026.05-2 | 0/2 each, in `GicV2ExitBootServicesEvent` | | `7af1d3307` | d3: `ID_AA64PFR0_EL1` read by the loader | 2024.11-5 | 4/4; `GIC` 0 under HVF, 1 under TCG | Oracles, each independent of the author: - CI's recorded failure. - UEFI 2.11 §2.3.6 and its attribute table. - The Arm ARM: an `_el1` encoding at EL2 is redirected to EL2's register when `E2H` is set, and `E2H` is RES0 without FEAT_VHE. - Linux v6.1's `efi_enter_kernel`, a differential implementation of the same handoff. - edk2's source and commits, and QEMU v11.1.1's (`hvf.c`, `virt.c`). - Each firmware binary, disassembled. ## Gates, at `b0e2f55d1` | gate | exit | |---|---| | `cargo run -- --ci host`: "Host: 56 step(s), all green"; the licence gate "6 exception(s) stand, and nothing else is refused" | 0 | | `cargo run -- --clippy`: "clippy: 17 invocations clean" | 0 | | `cargo run -- --build-only --arch aarch64` | 0 | | the same with `--kernel-feature boot-actuators --kernel-feature test-actuators` | 0 | | `cargo test --test toyos-build --no-run` | 0 | | C1, C2, D, n0, D with the refusal reverted, the EL1 arm leaving `M` set, and a measurement-only harness patch that boots Debian's 2026.05-2 for TCG `virt` guests, alone, with n0 and with D: each applied to `b0e2f55d1`, built both ways, the harness compiled, reverted, the tree clean. The patches are in a comment on this pull request. | 0 each | ## Unsure - No guest has booted `b0e2f55d1`. The whole suite, C1 and C2 under HVF, n0 and D on Debian's firmware and on Homebrew's, D with the refusal reverted, and the EL1 arm leaving `M` set are the orchestrator's runs. - The cleans of `KernelArgs` (C1) and of the handover's own instructions (C2) are new dependencies, and neither is measured. HVF is the only tier here with real caches. - `cortex-a72` is QEMU's TCG model; no silicon here runs it. - This branch's history keeps the deleted firmware, in `5f9508138`, `7af1d3307` and `ef659fc30`. Its objects take 4,383,389 bytes on disk beyond main's (`git rev-list --objects --disk-usage origin/main..HEAD`), and a merge commit carries them into main's history. ## Size `git diff --shortstat origin/main...HEAD`: 7 files, +176 −57. - Production: the loader +54 −12, the kernel +44 −30, `src/arch.rs` −2. - Tests: the harness +31 −13. - The issue +47. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- bootloader/src/arch/aarch64.rs | 66 ++++++++++++++---- ...608-hangs-at-exitbootservices-under-hvf.md | 47 +++++++++++++ kernel/src/arch/aarch64/boot.rs | 69 ++++++++++++------- kernel/src/arch/aarch64/control_regs.rs | 5 -- src/arch.rs | 2 - tests/common/qemu.rs | 25 +++++-- tests/toyos.rs | 19 ++--- 7 files changed, 176 insertions(+), 57 deletions(-) create mode 100644 issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md diff --git a/bootloader/src/arch/aarch64.rs b/bootloader/src/arch/aarch64.rs index a85bd2560b2..4b88b25c4b1 100644 --- a/bootloader/src/arch/aarch64.rs +++ b/bootloader/src/arch/aarch64.rs @@ -95,13 +95,20 @@ pub mod pio { } } -/// Hand the CPU to the kernel as firmware left it — its exception level, its -/// identity tables — at the image's physical entry, with `x0 = args`. The -/// kernel's entry switches to the boot map (`args.boot_pml4_addr`) itself (`kernel/src/arch/aarch64/boot.rs`), -/// because at EL2 only the kernel's own drop to EL1 can install it. +/// Hand the CPU to the kernel at the exception level firmware ran it at, with +/// that level's MMU and data cache off, at the image's physical entry with +/// `x0 = args`: the state PSCI's `CPU_ON` starts every other CPU in. Nothing +/// firmware left in a translation regime is walked past this point. UEFI +/// promises an identity map of RAM and none of its attributes (2.11 §2.3.6), +/// edk2's ArmVirtQemu maps `EfiLoaderData`, where this image is, execute-never +/// unless built otherwise, and the kernel's entry rewrites `HCR_EL2.E2H`, which +/// chooses the regime firmware's EL2 tables are walked in. That entry installs +/// the boot map (`args.boot_pml4_addr`) itself (`kernel/src/arch/aarch64/boot.rs`), +/// because at EL2 only its own drop to EL1 can. /// -/// The image is cleaned to the point of coherency first: that entry fetches -/// instructions with the MMU off for a few of them, straight from memory. +/// What runs or is read with the MMU off — the image, `args`, and this +/// function's own last instructions — is cleaned to the point of coherency +/// first. /// /// # Safety /// `args.boot_pml4_addr` is the boot map, `image` is the relocated kernel image, `entry_offset` @@ -109,20 +116,55 @@ pub mod pio { /// kernel copies it. pub unsafe fn enter_kernel(image: (u64, u64), entry_offset: u64, args: &KernelArgs) -> ! { write_back(image.0, image.1 as usize); - let entry = image.0 + entry_offset; + write_back(args as *const KernelArgs as u64, size_of::()); + let step = line(); // SAFETY: interrupts masked for good — the kernel's vectors are not - // installed yet — then every instruction cache line invalidated against - // the image just cleaned, and a branch to its entry with `x0 = args`, the - // boot protocol `toyos-abi::boot` and the kernel's `_start` define. + // installed yet. Every line from `2:` to `5:` is cleaned to the point of + // coherency, because those instructions are fetched with the MMU off, and + // every instruction cache line is invalidated against the image cleaned + // above. `SCTLR_ELx.M` and `.C` are cleared at the level this runs at, + // which fetches the next instruction from the same address because + // firmware's map is the identity. Then a branch to the image's entry with + // `x0 = args`, the boot protocol `toyos-abi::boot` and the kernel's + // `_start` define. `x9` and `x10` are scratch, and nothing returns to + // observe them. unsafe { core::arch::asm!( "msr daifset, #0xf", + "adr x9, 2f", + "bic x9, x9, x3", + "adr x10, 5f", + "1:", + "dc civac, x9", + "add x9, x9, x2", + "cmp x9, x10", + "b.lo 1b", + "dsb sy", "ic iallu", "dsb ish", "isb", - "br {entry}", - entry = in(reg) entry, + "2:", + "mrs x9, CurrentEL", + "cmp x9, #(2 << 2)", + "b.ne 3f", + "mrs x9, sctlr_el2", + "bic x9, x9, #(1 << 0)", + "bic x9, x9, #(1 << 2)", + "msr sctlr_el2, x9", + "b 4f", + "3:", + "mrs x9, sctlr_el1", + "bic x9, x9, #(1 << 0)", + "bic x9, x9, #(1 << 2)", + "msr sctlr_el1, x9", + "4:", + "isb", + "br x1", + "5:", in("x0") args as *const KernelArgs, + in("x1") image.0 + entry_offset, + in("x2") step, + in("x3") step - 1, options(noreturn), ); } diff --git a/issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md b/issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md new file mode 100644 index 00000000000..682ef9a4a7e --- /dev/null +++ b/issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md @@ -0,0 +1,47 @@ +--- +status: assigned +kind: tooling +opened: 2026-10-01 +--- + +# edk2 stable202502 to stable202608 hangs at ExitBootServices under HVF + +When a host's QEMU installation ships an edk2 from stable202502 to +stable202608, `virt` guests under HVF never return from `ExitBootServices`. +Homebrew's QEMU 11.1.1 ships stable202408 today. Owner: the orchestrator. + +Under QEMU 11.1.1's HVF a `virt` guest reads `ID_AA64PFR0_EL1.GIC` as 0, +though its GICv3's system registers answer. `hvf_arch_init_vcpu` +(`target/arm/hvf/hvf.c:1481`) writes that register once, setting `GIC` only if +`env->gicv3state` is set, and it runs while `machvirt_init` realizes the CPUs +(`hw/arm/virt.c:3140`), before `create_gic` (`:3158`) makes the GIC that sets +it. QEMU master is the same. TCG computes the field on each read +(`id_aa64pfr0_read`, `target/arm/helper.c`) and reads 1. The orchestrator's +`aarch64fw-r3-d3-pfr0` read the register from the loader: `0x1101000010110011`, +`GIC` 0, under HVF, and `0x1301001121110022` at EL1 and `0x1301001121110222` +at EL2, `GIC` 1, under TCG. + +From edk2-stable202502 (`8edd5fd6d3`, `eaa60a6b10`, `e663b79f74`) to +stable202608, ArmVirtQemu's `ArmGicDxe` runs its GICv3 driver only where that +field is nonzero, and its GICv2 driver otherwise. The DT's `arm,gic-v3` node +never sets the GICv2 CPU interface's base, which stays at its 0 default +(`ArmVirtQemu.dsc`). At `ExitBootServices`, `GicV2ExitBootServicesEvent` +acknowledges interrupts until `GICC_IAR` reads 1020 to 1023. At 0x0 + 0xc the +guest reads the firmware's own flash, `0xffffffff`, so the loop never ends. + +Seen at `8d74557fa` with Debian's 2026.05-2, in the orchestrator's runs +`aarch64fw-r2-d1-pinned` and `aarch64fw-r2-d2-pinned-no-vgic`. +`virt_early_panic` and `virt_early_fault`, this host's only HVF guests, stop +after the loader's last line. QMP then reads EL1h, PC `0xbf5e0a80` (the `ret` +after `MmioRead32`'s load, `X0 = 0xffffffff`) and `0xbf5dfbbc` a second later. +The 64 words dumped from 0x80 below the PC occur once in that build's +`ArmGicDxe.efi`, at file offset 0x1b3c: the loop above. With +`kernel-irqchip=off` (QEMU's own GICv3) the loop is the same. Every TCG `virt` +guest boots that build green. + +## Exit condition + +An edk2 release carrying `aefdbf91f4` and `377a890d80`, where the DT chooses +between split GICv2 and GICv3 drivers again, or a QEMU whose HVF sets +`ID_AA64PFR0_EL1.GIC` for an attached GICv3. It is shown by `virt_early_panic` +and `virt_early_fault` green under HVF on that firmware or that QEMU. diff --git a/kernel/src/arch/aarch64/boot.rs b/kernel/src/arch/aarch64/boot.rs index bdd84b1aa01..97f358f9f6a 100644 --- a/kernel/src/arch/aarch64/boot.rs +++ b/kernel/src/arch/aarch64/boot.rs @@ -2,20 +2,22 @@ //! to, and what `kernel_main` asks of this architecture at the points where //! one differs from another. //! -//! **The entries.** The loader leaves the boot CPU as firmware ran it — at EL2 -//! or EL1, on firmware's identity tables — cleans the kernel image and its own -//! tables to the point of coherency, and jumps to [`_start`]'s physical -//! address with `x0 = &KernelArgs`. PSCI starts every other CPU at -//! [`ap_start`]'s physical address, MMU off, at the level firmware gives an -//! operating system, with `x0` its [`ApStart`]'s physical address. Each names -//! a root table and branches to [`apply_declaration`], which writes the -//! [`control_regs`](super::control_regs) declaration whole: at EL2 it writes -//! `HCR_EL2` first, halts in a named refusal unless it reads back as declared, -//! then programs EL1's registers with the MMU already on and drops with -//! `ERET`; at EL1 it turns the MMU off first, so no translation register -//! changes under a live walk. Either way the entry resumes at its link address -//! in the view at `PHYS_OFFSET`, installs the vectors on its own stack, and -//! calls `kernel_main` or `smp::ap_entry`. +//! **The entries.** Every CPU arrives at its physical address with the MMU of +//! the level it runs at off, so nothing firmware left in a translation regime — +//! its tables, their execute-never attributes, the regime `HCR_EL2.E2H` chose — +//! is walked under either entry. The loader hands the boot CPU over at the +//! level firmware ran it, EL2 or EL1, with the kernel image and [`KernelArgs`] +//! cleaned to the point of coherency, and jumps to [`_start`] with +//! `x0 = &KernelArgs`. PSCI starts every other CPU at [`ap_start`], at the +//! level firmware gives an operating system, with `x0` its [`ApStart`]'s +//! physical address. Each names a root table and branches to +//! [`apply_declaration`], which writes the [`control_regs`](super::control_regs) +//! declaration whole: at EL2 it writes `HCR_EL2` first, halts in a named +//! refusal unless it reads back as declared, then programs EL1's registers with +//! the MMU already on and drops with `ERET`; at EL1 it programs them and turns +//! the MMU on. Either way the entry resumes at its link address in the view at +//! `PHYS_OFFSET`, installs the vectors on its own stack, and calls +//! `kernel_main` or `smp::ap_entry`. use core::mem::offset_of; @@ -28,8 +30,8 @@ use crate::drivers::acpi::direct_phys; use crate::log; use crate::mm::Region; -/// Entry point: the loader jumps here at its physical address, MMU on under -/// firmware's identity map, with `x0 = &KernelArgs`. +/// Entry point: the loader jumps here at its physical address with this +/// level's MMU off, and `x0 = &KernelArgs`. /// # Safety /// Only the loader may call this, fresh from firmware, with `x0` holding a live [`KernelArgs`]. #[unsafe(naked)] @@ -108,7 +110,10 @@ pub(super) unsafe extern "C" fn ap_start() -> ! { /// its physical address, and never returns. /// # Safety /// Only [`_start`] and [`ap_start`] branch here, with `x3` a root that maps the -/// kernel image at both its physical and its link address. +/// kernel image at both its physical and its link address. The EL1 arm's +/// translation registers and the EL2 arm's `HCR_EL2` change under no walk only +/// because the level entered at has its MMU off; an entry with it on halts in +/// [`refused_mmu_on_at_entry`]. #[unsafe(naked)] unsafe extern "C" fn apply_declaration() -> ! { core::arch::naked_asm!( @@ -125,10 +130,9 @@ unsafe extern "C" fn apply_declaration() -> ! { "b.eq 2f", "cmp x21, #1", "b.ne 9f", - // EL1: translation off, then the declaration, then translation on. - "ldr x1, ={sctlr_off}", - "msr sctlr_el1, x1", - "isb", + // EL1: refused with its MMU on; else the declaration, then translation on. + "mrs x1, sctlr_el1", + "tbnz x1, #0, {refuse_mmu}", "msr mair_el1, x4", "msr tcr_el1, x2", "msr ttbr0_el1, x3", @@ -143,10 +147,14 @@ unsafe extern "C" fn apply_declaration() -> ! { "msr sctlr_el1, x5", "isb", "br x22", - // EL2: `HCR_EL2` first, since with `E2H` set every `_el1` name - // below is an EL2 register; refused unless it reads back as declared. - // Then EL1's registers with the MMU on, EL2's others, and the drop. + // EL2: refused with its own MMU on, so the regime `E2H` chooses + // changes under no walk. Then `HCR_EL2`, since with `E2H` set every + // `_el1` name below is an EL2 register; refused unless it reads back + // as declared. Then EL1's registers with the MMU on, EL2's others, and + // the drop. "2:", + "mrs x1, sctlr_el2", + "tbnz x1, #0, {refuse_mmu}", "ldr x1, ={hcr}", "msr hcr_el2, x1", "isb", @@ -194,8 +202,8 @@ unsafe extern "C" fn apply_declaration() -> ! { ips = const regs::TCR_IPS_SHIFT, mair = const regs::MAIR, sctlr = const regs::SCTLR, - sctlr_off = const regs::SCTLR_MMU_OFF, hcr = const regs::HCR_EL2, + refuse_mmu = sym refused_mmu_on_at_entry, refuse_hcr = sym refused_hcr_el2_readback, cnthctl = const regs::CNTHCTL_EL2, cptr = const regs::CPTR_EL2, @@ -206,6 +214,17 @@ unsafe extern "C" fn apply_declaration() -> ! { ); } +/// Where a CPU halts that reaches [`apply_declaration`] with `SCTLR_ELx.M` set +/// at the level it runs at, as one a loader hands over under firmware's tables +/// does: the declaration's translation registers and `HCR_EL2` would change +/// under a live walk. Nothing can report yet, so the refusal is this symbol, +/// which the halted PC names. +#[unsafe(naked)] +#[no_mangle] +unsafe extern "C" fn refused_mmu_on_at_entry() -> ! { + core::arch::naked_asm!("1:", "wfe", "b 1b") +} + /// Where a CPU entered at EL2 halts when `HCR_EL2` reads back anything but the /// declaration the entry wrote: `E2H` held set, which the loader refuses by /// name first (`bootloader/src/arch/aarch64.rs`), or any other bit. There the diff --git a/kernel/src/arch/aarch64/control_regs.rs b/kernel/src/arch/aarch64/control_regs.rs index 0cad5e024e2..6a652ab9a64 100644 --- a/kernel/src/arch/aarch64/control_regs.rs +++ b/kernel/src/arch/aarch64/control_regs.rs @@ -26,11 +26,6 @@ pub use toyos_bootmap::aarch64::MAIR; /// levels; EL0's cache maintenance and `WFI`/`WFE` trap. pub const SCTLR: u64 = SCTLR_RES1 | 1 << 0 | 1 << 2 | 1 << 3 | 1 << 4 | 1 << 7 | 1 << 8 | 1 << 12; -/// `SCTLR_EL1` with the MMU and caches off — [`SCTLR`] less `M`, `C`, `I`, -/// `SA` and `SA0` — the only other value the entry writes: what a CPU entered at EL1 under firmware's tables is put through -/// before its translation registers change. -pub const SCTLR_MMU_OFF: u64 = SCTLR_RES1 | 1 << 7 | 1 << 8; - const SCTLR_RES1: u64 = 1 << 29 | 1 << 28 | 1 << 23 | 1 << 22 | 1 << 20 | 1 << 11; /// `TCR_EL1` but for `IPS`: 48-bit regions from both tables (`T0SZ` = `T1SZ` = diff --git a/src/arch.rs b/src/arch.rs index d93bc9fd6c8..049d3154aed 100644 --- a/src/arch.rs +++ b/src/arch.rs @@ -155,8 +155,6 @@ impl Arch { } } - /// The CPU every guest of this architecture gets under `accel`. - /// /// **One declaration, read by `cargo run` and by the harness both**, because /// the two drifted: the harness gained `+smep` and the interactive path did /// not, so the machine an owner looked at differed from the machine the diff --git a/tests/common/qemu.rs b/tests/common/qemu.rs index 93952d7b541..577e080b9d2 100644 --- a/tests/common/qemu.rs +++ b/tests/common/qemu.rs @@ -740,6 +740,11 @@ pub enum Profile { /// firmware then hands the loader the CPU at EL2, and the kernel's entry /// has to drop from it. HVF gives a guest EL1 only. VirtEl2, + /// [`Profile::VirtEl2`] on `-cpu cortex-a72`, which has no FEAT_VHE: any + /// firmware hands the loader EL2 with `HCR_EL2.E2H` clear, where an `_el1` + /// register name is EL1's own, so the loader's EL2 arm alone turns EL2's + /// MMU off. + VirtEl2NoVhe, /// [`Profile::Virt`] emulated on `-cpu max` whatever the host: firmware /// hands the loader the CPU at EL1, as HVF does, and QEMU's FADT names /// PSCI's conduit `HVC`; unlike HVF's, the CPU has RNDR for the kernel's @@ -751,7 +756,7 @@ impl Profile { /// The architecture this machine is. pub fn arch(self) -> Arch { match self { - Self::Virt | Self::VirtEl2 | Self::VirtTcg => Arch::Aarch64, + Self::Virt | Self::VirtEl2 | Self::VirtEl2NoVhe | Self::VirtTcg => Arch::Aarch64, Self::Headless | Self::HeadlessNoIommu | Self::Gop @@ -762,10 +767,18 @@ impl Profile { /// How this host provides the machine. pub fn accel(self) -> Accel { match self { - Self::VirtEl2 | Self::VirtTcg => Accel::Tcg, + Self::VirtEl2 | Self::VirtEl2NoVhe | Self::VirtTcg => Accel::Tcg, _ => self.arch().accel(), } } + + /// The CPU this machine has. + fn cpu(self) -> &'static str { + match self { + Self::VirtEl2NoVhe => "cortex-a72", + _ => self.arch().cpu(self.accel()), + } + } } /// The vIOMMU a profile puts on the machine. @@ -884,7 +897,7 @@ pub const NVME_SMALL: u64 = 128 * 1024 * 1024; impl Profile { fn shape(self) -> Shape { match self { - Self::VirtEl2 | Self::VirtTcg => Self::Virt.shape(), + Self::VirtEl2 | Self::VirtEl2NoVhe | Self::VirtTcg => Self::Virt.shape(), Self::Virt => Shape { vga: "std", panel: None, @@ -1945,7 +1958,9 @@ fn qemu_command( // The unit a profile declares is VT-d, which `virt` has none of. assert!(shape.iommu.is_none(), "`virt` has no VT-d"); match options.profile { - Profile::VirtEl2 => format!("{},gic-version=3,virtualization=on", arch.machine()), + Profile::VirtEl2 | Profile::VirtEl2NoVhe => { + format!("{},gic-version=3,virtualization=on", arch.machine()) + } _ => format!("{},gic-version=3", arch.machine()), } } @@ -1964,7 +1979,7 @@ fn qemu_command( qemu.arg("-machine") .arg(&machine) .arg("-cpu") - .arg(arch.cpu(accel)) + .arg(options.profile.cpu()) .arg("-smp") .arg(options.smp.to_string()) .arg("-m") diff --git a/tests/toyos.rs b/tests/toyos.rs index 13a0f210766..170e292f4b0 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -135,7 +135,7 @@ const SCREEN_TESTS: &[(&str, qemu::Profile)] = &[ ("screen_fatal_halt_composited", qemu::Profile::Metal), ("virt_early_panic", qemu::Profile::Virt), ("virt_early_fault", qemu::Profile::Virt), - ("virt_el2_drop", qemu::Profile::VirtEl2), + ("virt_el2_drop", qemu::Profile::VirtEl2NoVhe), ("virt_user_mode", qemu::Profile::VirtEl2), ("virt_timer_preempts", qemu::Profile::VirtEl2), ("virt_irq_storm", qemu::Profile::VirtEl2), @@ -1605,12 +1605,15 @@ fn run_screen_test(name: &str, profile: qemu::Profile, test_config: &Path) -> Re } "virt_el2_drop" => { // The entry's drop from EL2, which HVF never exercises: `virt` with - // EL2 under TCG, where firmware hands the loader the CPU at EL2. A - // loader that refuses the CPU says so and stops; a drop that leaves - // `HCR_EL2` other than declared halts in a named refusal and says - // nothing; one that lands anywhere but EL1 on `SP_EL1` panics in the - // declaration's read-back. Each way the line this waits for never - // comes. + // EL2 under TCG, where firmware hands the loader the CPU at EL2, on + // a CPU without FEAT_VHE, so `E2H` is clear at the handover and + // only the loader's EL2 arm turns EL2's MMU off. A loader that + // refuses the CPU says so and stops; a handover with EL2's MMU on + // halts in a named refusal, or faults first where firmware maps the + // image execute-never; a drop that leaves `HCR_EL2` other than + // declared halts in a named refusal and says nothing; one that lands + // anywhere but EL1 on `SP_EL1` panics in the declaration's + // read-back. Each way the line this waits for never comes. let mut qemu = QemuInstance::boot_with_options( test_config, &[], @@ -1625,7 +1628,7 @@ fn run_screen_test(name: &str, profile: qemu::Profile, test_config: &Path) -> Re let rest = qemu.drain_until(Duration::from_secs(10), |l| l.contains(EARLY_PANIC_MESSAGE)); let serial = format!("{}\n{rest}", qemu.boot_log()); for want in [ - "CPU: entered at EL2, HCR_EL2.E2H ", + "CPU: entered at EL2, HCR_EL2.E2H 0,", "ID_AA64MMFR4_EL1.E2H0 0x0: the kernel's entry writes E2H clear", "as declared; entered at EL2, HCR_EL2 read back as declared", "EARLY PANIC: panicked at", From f182b6cfad1942515e25e4ffd26f99cddab16f97 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 20:42:04 +0200 Subject: [PATCH 2/2] The AArch64 loader clears SCTLR_ELx.M alone at the handover Both `bic x9, x9, #(1 << 2)` in `enter_kernel` go, with the prose that named `C`: "and data cache" and the PSCI `CPU_ON` clause in its doc, and `.C` in its SAFETY comment. With `M` clear, `C` governs no access, so clearing it bought nothing. The Arm ARM's shared pseudocode (DDI 0487), as Arm's 2023-03 machine-readable release prints it (ISA_A64_xml_A_profile-2023-03, shared_pseudocode.xml): `AArch64.S1Translate` returns `AArch64.S1DisabledOutput` whenever `AArch64.S1Enabled` is false, which is `SCTLR_ELx.M` clear or, at EL1&0, `HCR_EL2.DC` or `.TGE` set. `S1DisabledOutput` gives a data access Device-nGnRnE, or Normal Inner and Outer Write-Back at EL1&0 where `HCR_EL2.DC` is 1, and an instruction fetch Normal memory cacheable by `SCTLR_ELx.I`; it reads `C` for none of them. `AArch64.S1DCacheEnabled`, the only reader of `C`, has two callers, both on paths with stage 1 enabled: `S1Translate` after its walk, and `S1Walk`. `SCTLR_EL1.C`'s own description (SysReg_xml_A_profile-2023-03, AArch64-sctlr_el1.xml): "When the Effective value of the HCR_EL2.DC bit in the current Security state is 1, the PE ignores SCTLR_EL1.C." Past the handover nothing reads the `C` firmware left: the kernel writes `SCTLR_EL1` whole as its MMU comes on, and never writes `SCTLR_EL2`. The review's REMOVEs go with it: the HVF hazard issue's two sentences on what Homebrew's QEMU and QEMU master ship today, and `virt_el2_drop`'s clause restating `Profile::VirtEl2NoVhe`'s doc. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- bootloader/src/arch/aarch64.rs | 15 ++++++--------- ...02608-hangs-at-exitbootservices-under-hvf.md | 4 ++-- tests/toyos.rs | 17 ++++++++--------- 3 files changed, 16 insertions(+), 20 deletions(-) diff --git a/bootloader/src/arch/aarch64.rs b/bootloader/src/arch/aarch64.rs index 4b88b25c4b1..0c98d759110 100644 --- a/bootloader/src/arch/aarch64.rs +++ b/bootloader/src/arch/aarch64.rs @@ -96,10 +96,9 @@ pub mod pio { } /// Hand the CPU to the kernel at the exception level firmware ran it at, with -/// that level's MMU and data cache off, at the image's physical entry with -/// `x0 = args`: the state PSCI's `CPU_ON` starts every other CPU in. Nothing -/// firmware left in a translation regime is walked past this point. UEFI -/// promises an identity map of RAM and none of its attributes (2.11 §2.3.6), +/// that level's MMU off, at the image's physical entry with `x0 = args`. +/// Nothing firmware left in a translation regime is walked past this point. +/// UEFI promises an identity map of RAM and none of its attributes (2.11 §2.3.6), /// edk2's ArmVirtQemu maps `EfiLoaderData`, where this image is, execute-never /// unless built otherwise, and the kernel's entry rewrites `HCR_EL2.E2H`, which /// chooses the regime firmware's EL2 tables are walked in. That entry installs @@ -122,9 +121,9 @@ pub unsafe fn enter_kernel(image: (u64, u64), entry_offset: u64, args: &KernelAr // installed yet. Every line from `2:` to `5:` is cleaned to the point of // coherency, because those instructions are fetched with the MMU off, and // every instruction cache line is invalidated against the image cleaned - // above. `SCTLR_ELx.M` and `.C` are cleared at the level this runs at, - // which fetches the next instruction from the same address because - // firmware's map is the identity. Then a branch to the image's entry with + // above. `SCTLR_ELx.M` is cleared at the level this runs at, which + // fetches the next instruction from the same address because firmware's + // map is the identity. Then a branch to the image's entry with // `x0 = args`, the boot protocol `toyos-abi::boot` and the kernel's // `_start` define. `x9` and `x10` are scratch, and nothing returns to // observe them. @@ -149,13 +148,11 @@ pub unsafe fn enter_kernel(image: (u64, u64), entry_offset: u64, args: &KernelAr "b.ne 3f", "mrs x9, sctlr_el2", "bic x9, x9, #(1 << 0)", - "bic x9, x9, #(1 << 2)", "msr sctlr_el2, x9", "b 4f", "3:", "mrs x9, sctlr_el1", "bic x9, x9, #(1 << 0)", - "bic x9, x9, #(1 << 2)", "msr sctlr_el1, x9", "4:", "isb", diff --git a/issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md b/issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md index 682ef9a4a7e..0888190aeb2 100644 --- a/issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md +++ b/issues/build/edk2-stable202502-to-202608-hangs-at-exitbootservices-under-hvf.md @@ -8,14 +8,14 @@ opened: 2026-10-01 When a host's QEMU installation ships an edk2 from stable202502 to stable202608, `virt` guests under HVF never return from `ExitBootServices`. -Homebrew's QEMU 11.1.1 ships stable202408 today. Owner: the orchestrator. +Owner: the orchestrator. Under QEMU 11.1.1's HVF a `virt` guest reads `ID_AA64PFR0_EL1.GIC` as 0, though its GICv3's system registers answer. `hvf_arch_init_vcpu` (`target/arm/hvf/hvf.c:1481`) writes that register once, setting `GIC` only if `env->gicv3state` is set, and it runs while `machvirt_init` realizes the CPUs (`hw/arm/virt.c:3140`), before `create_gic` (`:3158`) makes the GIC that sets -it. QEMU master is the same. TCG computes the field on each read +it. TCG computes the field on each read (`id_aa64pfr0_read`, `target/arm/helper.c`) and reads 1. The orchestrator's `aarch64fw-r3-d3-pfr0` read the register from the loader: `0x1101000010110011`, `GIC` 0, under HVF, and `0x1301001121110022` at EL1 and `0x1301001121110222` diff --git a/tests/toyos.rs b/tests/toyos.rs index 170e292f4b0..24ca74784cb 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -1605,15 +1605,14 @@ fn run_screen_test(name: &str, profile: qemu::Profile, test_config: &Path) -> Re } "virt_el2_drop" => { // The entry's drop from EL2, which HVF never exercises: `virt` with - // EL2 under TCG, where firmware hands the loader the CPU at EL2, on - // a CPU without FEAT_VHE, so `E2H` is clear at the handover and - // only the loader's EL2 arm turns EL2's MMU off. A loader that - // refuses the CPU says so and stops; a handover with EL2's MMU on - // halts in a named refusal, or faults first where firmware maps the - // image execute-never; a drop that leaves `HCR_EL2` other than - // declared halts in a named refusal and says nothing; one that lands - // anywhere but EL1 on `SP_EL1` panics in the declaration's - // read-back. Each way the line this waits for never comes. + // EL2 under TCG, where firmware hands the loader the CPU at EL2. A + // loader that refuses the CPU says so and stops; a handover with + // EL2's MMU on halts in a named refusal, or faults first where + // firmware maps the image execute-never; a drop that leaves + // `HCR_EL2` other than declared halts in a named refusal and says + // nothing; one that lands anywhere but EL1 on `SP_EL1` panics in the + // declaration's read-back. Each way the line this waits for never + // comes. let mut qemu = QemuInstance::boot_with_options( test_config, &[],