The ITS under HVF: one declaration picks each machine's interrupt controller, and virt_claim_lpi is back

[Japabu]

First of the two PRs that redo the closed #869: the GIC ITS under HVF alone. Aperture and ECAM placement are the second, on another branch. This takes up issues/virt-claim-lpi-boot-stalled-in-firmware-once-under-load.md.

Problem

cargo run -- --arch aarch64 gave virt an SMMUv3 but no ITS. QEMU 11.1 runs HVF guests on Apple's in-kernel vGIC by default. That vGIC offers GICv2m and refuses an ITS, so the kernel refused every PCI claim on the interactive machine because its interrupts could not be remapped. The harness had its own spelling for its one ITS machine, its=on,kernel-irqchip=off. That spelling applied under every accelerator, but KVM cannot run a GICv3 with its irqchip off. The two launchers had drifted apart, as #869's review pointed out.

Change

• Arch::irqchip(accel, remapped) sits beside Arch::cpu(accel) and is the one declaration that cargo run (build/qemu.rs) and the harness (tests/common/qemu.rs) both read:
  • x86-64: VT-d's kernel-irqchip=split. The machine strings are unchanged.
  • AArch64: gic-version=3. With remapped it adds the ITS through msi=its (the its property is deprecated). Under HVF alone it also adds kernel-irqchip=off, so QEMU's own GIC is used. KVM keeps its in-kernel GICv3, and TCG emulates the ITS either way.
• virt_claim_lpi is back: b65ba04 is reverted onto the moved paths (tests/images/virtclaimcase.toml, tests/guest/rust-tests/src/bin/claim_*.rs). It is the only test of the ITS path: a claimed function's LPI, the claim the IORT routes past the SMMUv3, and a fault in a claimed BAR. It needs a guest because these are the live kernel's ITS commands and QEMU's IORT/SMMUv3 routing. Its machine is Profile::VirtIts: HVF on an Apple host, where it was booted; TCG on other hosts and KVM on a Linux arm64 host whose /dev/kvm opens, neither booted here (QEMU's TCG info qtree builds the same GIC, ITS and SMMUv3).
• issues/an-aarch64-cargo-run-boot-hands-over-no-pci-claim.md (from main) no longer says the interactive machine lacks an ITS; it keeps the BAR-placement failure that PR2 takes up.

Decisions and tradeoffs

• HVF gives up its hardware vGIC on machines that claim PCI functions. QEMU then handles interrupts in userspace on cargo run's AArch64 machine. No speed claim is made either way. The alternative is no isolated claims at all.
• The firmware stall stays open. The test's one red was a firmware stall before the loader. About 4,000 deliberate boots never reproduced it. I restore the test because it is the only coverage of the ITS path, and the stall is not a defect in the test. The issue keeps its reproduction exit. It now records that cargo run's HVF machine runs on the same GIC, so if the stall comes from that GIC, it can reach the interactive machine too. I did not attempt another reproduction batch.
• The interactive machine is still not fully working. With an ITS, a claim now gets past interrupt remapping. Codex's boot at 6fb2ba893 (its=on spelling, not rerun here) then found BAR placement exhausted inside ECAM. Fixing that is PR2.

Checks (host: 14-core Apple Silicon, macOS, QEMU 11.1.1, HVF)

• cargo test --test toyos-build -- virt_claim_lpi: exit 0, at load 30→58. The ITS was armed from the MADT, edu was handed over on slot 0 with LPI 8192 and DeviceID 0x18, 16 messages were read as 16 interrupts, e1000e was refused with NoDeviceId, and two BAR faults were contained.
• cargo test --test toyos-build -- virt_smmu virt_low_ecam virt_user_mode virt_el2_drop iommu_virtio_platform: exit 0. These are the neighbours whose -machine construction was rewritten, covering HVF, TCG at EL2, and x86 with and without VT-d.
• Negative control (negative-control.patch, applied, built, then reversed by an EXIT trap):
  • The patch gives the HVF arm the base's machine (gic-version=3) and drops the test's msi=its argv check.
  • virt_claim_lpi went red with exit 1.
  • The kernel logged ITS: the MADT names none, and every claim was refused: "its interrupts would not be remapped on this machine". This is the refusal the interactive machine had.
  • The load was about 37, but the red does not depend on load.
• Independent oracle (QEMU's own info qtree under HVF):
  • gic-version=3,msi=its,kernel-irqchip=off,iommu=smmuv3 builds arm-gicv3 and arm-gicv3-its.
  • Without kernel-irqchip=off, QEMU exits: "ITS not supported on HVF when using the hardware vGIC".
  • gic-version=3 alone builds hvf-arm-gicv3 with arm-gicv2m.
  • Under TCG, msi=its builds the ITS.
  • KVM was not measured. The source is QEMU v11.1.1 hw/arm/virt.c: finalize_gic_version allows only GICv2 when KVM's irqchip is off, and finalize_msi_controller handles the HVF/ITS case.
• cargo run -- --ci host: exit 0, all 76 steps green.
• I did not run --build-only, because no image content changed.

Evidence: ToyOSOrg/evidence pr-912 at 902ed2117, its-7272d0e8a/. Tested revision: every log ran on the tree staged on ef684e4fc that was committed unchanged as 7272d0e8a, apart from a two-line prose reflow in an issue (REVISION there). aa4104583 merges main and corrects two issues' prose only, so no rerun applies.

Lines: build/arch.rs +17, build/qemu.rs +8 −9. tests/common/qemu.rs is +39 −23, of which the restored VirtIts profile accounts for +22 −1 and the shared machine construction for the rest. The restored test and its binaries and image are +387, which matches b65ba04's deletion. The issues are +19 −24.

🤖 Generated with Claude Code

https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy

[Japabu]

Review: #912 at 7272d0e8a, against origin/main (merge base ef684e4fc; git merge-tree with ed1ec1512 is clean).

Architecture

This matches design point 1 of #869 as the owner ruled it: an ITS under HVF alone, in one declaration that both launchers read, and shapes that do not ask for remapping keep their controller. Arch::irqchip(accel, remapped) sits beside Arch::cpu(accel), and build/qemu.rs and tests/common/qemu.rs both read it. Only the HVF arm adds kernel-irqchip=off. #869's blocker (kernel-irqchip=off under KVM) is gone.

x86-64 strings. I checked these by hand.

• cargo run: q35,kernel-irqchip=split before and after. Every interactive shape has iommu: true.
• Harness: before, split was added for any VT-d unit; now it is added when unit.intremap is set. Every Unit::Vtd in the tree is IOMMU_DEFAULT (intremap: true). AMD-Vi gets no split in either version.

So the strings are byte-identical today. The new condition is also the more accurate one. The non-ITS AArch64 strings keep their option order.

The restored test. I diffed the three claim_* binaries and virtclaimcase.toml against b65ba04fe^: they are identical. tests/toyos.rs differs in two places. Paths are moved, which drops the config.parent() line. The argv check now looks for msi=its instead of its=on/kernel-irqchip=off. HVF needs kernel-irqchip=off, so leaving it out of the check is right. The build/build.rs config list and the RUST_SKIP entries are restored.

Credit to Codex. The commit message credits 1c582085d (build/qemu.rs) and the #869 review that asked for the per-accelerator declaration. What was taken is exactly the idea of QEMU's GIC plus its ITS on the interactive HVF machine. The code was rewritten. The credit is accurate.

Cost of kernel-irqchip=off. The PR body states it honestly: the hardware vGIC is given up for cargo run's AArch64 machine, and no speed claim is made. The owner's ruling covers this choice. issues/toyos-runs-on-arm64.md:370-373 records which GIC is used and why. No correctness or audio/visual quality is weakened, so no issue is owed.

Negative control. The whole-change revert deletes the test, so it cannot serve as the control. The patch instead gives the HVF arm the base's interactive spelling (gic-version=3) and drops the argv check. That is the right control for two claims:

• the HVF arm is what gives the machine its ITS;
• the kernel-line assertions alone detect its absence.

negative-control.log shows ITS: the MADT names none and claims refused "its interrupts would not be remapped on this machine", with exit 1. The EXIT trap restored the tree (RESTORED 0). The saved arch.rs.orig is identical to the committed build/arch.rs.

The QEMU oracle. info qtree is an independent check of what each machine string builds. It used cargo run's exact machine string.

The stall issue. Its exit is still honest: reproduce the stall with a deliberate harness, name the cause, fix it. The issue still records that about 4,000 boots did not reproduce it. It now states the remaining risk to the interactive machine as a conditional ("if it is that GIC's"). Restoring the test while the stall stays open is justified: the stall is in firmware, before the loader, and is independent of what the test claims. The test is the only coverage of the ITS path. This answers the owner's condition that the change "accounts for the recorded firmware stall on that configuration".

Size. +470 −56. Build production code is about +25 −9. The restored test, its binaries and its image are +387, matching b65ba04fe. The issues are +19 −24.

Findings

BLOCKER

• issues/an-aarch64-cargo-run-boot-hands-over-no-pci-claim.md:9-13: this change makes the paragraph false.

  • It says cargo run -- --arch aarch64 launches virt,gic-version=3,iommu=smmuv3, that HVF's GIC has no ITS, and that every claim is refused for want of remapping.
  • After this PR, build/qemu.rs launches virt,gic-version=3,msi=its,kernel-irqchip=off,iommu=smmuv3. The issue's remaining defect is the BAR placement in lines 15ff, which is PR2's.
  • This PR fixes the first of the issue's two causes, so it takes that half up and must correct it. Otherwise the next agent starts PR2 from a false account of the interactive machine.
  • Fix: replace lines 9-17 with one sentence. cargo run's machine carries the ITS (Arch::irqchip), and each claim's DeviceID is mapped. The hand-run log at fix2-60f08b937 is the evidence that the ITS spelling arms it. Keep the BAR-placement account as it is.

• Evidence (orchestrator's to resolve):

  • It is not published. The PR cites a local scratch directory nobody else can read.
  • None of the logs records the tested revision. negative-control.log says HEAD ef684e4fc (index + this patch). claim-lpi-1.log and neighbours-1.log name no revision. The runs (20:58 to 21:05) came before the commit (21:27). ci-host.log started at 21:06.
  • My inference from mtimes: no changed source file was modified after 20:58:50 except build/arch.rs and tests/toyos.rs. Both were rewritten at 21:05:56 by the control's reversal, and arch.rs.orig equals the commit's file. The issue edits came last, at 21:06:31.
  • So the logs very probably test exactly 7272d0e8a's tree. That is still my inference, not something the evidence records.
  • Fix: publish the evidence, and add a note to it and to the PR body. The note should say the runs were on the uncommitted tree at ef684e4fc that was committed unchanged as 7272d0e8a, and why. No rerun is needed for this.

SUGGESTION

• The PR body declares Profile::VirtIts for "HVF on an Apple host and TCG elsewhere". Only HVF was booted.

  • The merge queue's guest / suite (x86 runner, TCG) will be the first run of virt,gic-version=3,msi=its,iommu=smmuv3 with this guest.
  • The oracle shows TCG builds the same arm-gicv3/arm-gicv3-its/arm-smmuv3 set. Its last line also shows TCG virt has an ITS even without msi=its. So the risk is low, and a failure would not land unseen.
  • The cheapest local run is a temporary patch, like the control's, that returns Accel::Tcg from Profile::accel for VirtIts.
  • "TCG elsewhere" also leaves out KVM on a Linux arm64 host whose /dev/kvm opens.

• tests/toyos.rs:2422: the argv check repeats what the kernel-line assertions already catch. The negative control had to delete it to reach ITS: 0 at 0x8080000 armed, which fails without an ITS. It is restored from the deleted test and only gives a clearer message for a harness misconfiguration. Keep it or drop it.

• issues/virt-claim-lpi-boot-stalled-in-firmware-once-under-load.md:16-17: two details are now stale.

  • It calls VirtIts "the one HVF profile on QEMU's own GICv3 (its=on,kernel-irqchip=off)". The spelling is now msi=its.
  • cargo run's machine now shares that GIC.

  This is historical context for the red, so fixing it is optional. The exit's harness boots the current spelling.

• PR body (orchestrator's): it says "at load 25→58", but claim-lpi-1.log reads 30.11 → 58.42.

LAND is withheld until the stale issue paragraph is corrected and the evidence is published with its revision note.

SEND BACK

[Japabu]

Evidence published by the orchestrator (the implementer's push was refused by a permission check, which the owner called odd and overrode): ToyOSOrg/evidence branch pr-912 at 24c3bddc218d52f1d927804dc3c6411c49e26f0b, directory its-7272d0e8a/, masked logs as the body describes. The body's "unpublished" line is the implementer's to replace with this link in its next push.

[Japabu]

Review update for #912 at 7272d0e8a: the evidence is now published.

The evidence is on ToyOSOrg/evidence branch pr-912, commit 24c3bddc2, under its-7272d0e8a/. I compared the published files with the copy I reviewed: all seven blob hashes match. My assessment of the logs is unchanged.

Evidence BLOCKER, revised:

• Unpublished: resolved. The body still says the evidence is not yet on the evidence repository. That line is now false and should point to pr-912@24c3bddc2/its-7272d0e8a/.
• Tested revision: still owed, and cheap. The published logs still do not name the revision they tested:
  • negative-control.log says HEAD ef684e4fc (index + this patch).
  • The other logs name no revision.
  • The runs (20:58 to 21:05) came before the commit (21:27).
  • The fix is the note I asked for: the runs were on the uncommitted tree at ef684e4fc that was committed unchanged as 7272d0e8a. My mtime check supports this, but the record must say it. No rerun is needed.

Still open: the stale-issue BLOCKER at issues/an-aarch64-cargo-run-boot-hands-over-no-pci-claim.md:9-13. The SUGGESTIONs are unchanged.

SEND BACK