Skip to content

Restore the guest X8 when the shim drops its frame #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jserv merged 1 commit into from
Sep 29, 2026
Merged

Restore the guest X8 when the shim drops its frame #383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions .claude/skills/elfuse-guest-abi/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,21 +63,25 @@ the block size the shim assumes.
|----|------|-------------|
| 0 | TLBI_NONE | skip flush |
| 1 | TLBI_BROADCAST | TLBI VMALLE1IS + DSB ISH + ISB |
| 2 | drop-frame | host rebuilt EL0 state; discard saved frame on ERET |
| 2 | drop-frame | host rebuilt EL0 state; discard saved frame on ERET, restoring no register except X8, reloaded from the frame's own X8 slot (`[sp, #64]`) |
| 3 | TLBI_RANGE | loop TLBI VAE1IS, X9=VA, X10=page count |
| 4 | TLBI_RANGE_LARGE | single RVAE1IS, encoded operand in X9 |

X11=1 is the icache-flush hint: set when a page transitions to executable, and
the shim then issues an IC invalidate alongside whichever TLBI it picked. The
shim restores X11 from the saved frame before ERET, so EL0 never observes it.
restoring tails reload X11 from the saved frame before ERET, so EL0 never
observes it. The X8 = 2 tail restores nothing but X8, so a delivery that follows
a page-table-modifying syscall in the same epilogue hands X9 through X11 out to
EL0 in place of the guest's.

X7 is the ptrace-stop request on the same return, and it obeys a rule the TLBI
codes do not. The shim reads it only after restoring the saved frame, so the
tracer snapshots the guest's architectural registers rather than shim scratch;
non-zero means take HVC #13 before the ERET. That makes X7 unusable on the one
tail that never restores the frame, X8 = 2, where the live registers already
are the final EL0 state and a host write to X7 would land in EL0 as guest
state. The host takes that stop inline in the epilogue instead and leaves X7
tail that never restores the frame, X8 = 2, where the live registers are the
final EL0 state with one exception -- X8 holds the marker, which is why that
tail reloads it from the frame -- and a host write to X7 would land in EL0 as
guest state. The host takes that stop inline in the epilogue instead and leaves X7
alone, and an `execve` re-entry, which has no tail at all, leaves the stop owed
for the new image.

Expand Down Expand Up @@ -123,7 +127,13 @@ your path goes through the dispatch epilogue at all:
- Inside the epilogue, set X8=2. The shim reads it as the drop-frame marker
and discards the saved GPR frame. Signal delivery on the syscall-return path
works this way. It does not need the marker when EL0 was preempted rather
than returning from a syscall, because there is no shim frame to drop.
than returning from a syscall, because there is no shim frame to drop. If
the X8 you want EL0 to see differs from the one the frame was entered with,
publish it into the frame's X8 slot with the marker: the marker occupies the
register, and the tail reloads X8 from that slot alone. Bound that write at
the EL1 stack region (`thread_sp_el1_region`), not at the shim data block:
the block's low end is the shim-globals cache, and only its top
`MAX_THREADS` slots are stack.
- Bypass the epilogue by returning `SYSCALL_EXEC_HAPPENED`. The epilogue
returns early, before it writes X0 or X8. `sys_execve` works this way, and
the normal X0 writeback is exactly what it needs to avoid. It must also skip
Expand Down
79 changes: 47 additions & 32 deletions docs/internals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,16 +285,15 @@ X8 == 1 TLBI_BROADCAST TLBI VMALLE1IS + DSB ISH + ISB
-> restore GPRs (keep X0); ERET
X8 == 2 drop-frame discard the saved GPR frame
(`add sp, sp, #256`) and ERET on the rebuilt
EL0 register state. Set by `execve` and
`rt_sigreturn` (which write the whole frame
directly into the vCPU) and by
`signal_deliver()` on the syscall-return
path (so handler PC/SP/LR/args installed by
the host are not overwritten by the stale
shim frame on ERET). `execve` additionally
issues `IC IALLU` because the new program
text may live in pages that previously held
the old text.
EL0 register state, reloading only `X8` from
the frame's own slot (`[sp, #64]`), since the
marker occupies that register. Set by
`rt_sigreturn` (which writes the whole
register set directly into the vCPU) and by
signal delivery on the syscall-return path
(so handler PC/SP/LR/args installed by the
host are not overwritten by the stale shim
frame on ERET). Always issues `IC IALLU`.
X8 == 3 TLBI_RANGE loop TLBI VAE1IS over `X9` (start VA),
`X10` (page count); 4 KiB granule. Used for
up to `TLBI_SELECTIVE_MAX_PAGES = 16` pages.
Expand All @@ -317,25 +316,40 @@ separate broadcast after the split lands.
`X8 == 2` is the generic drop-saved-frame marker: the host has
rebuilt EL0 register state directly into the vCPU and the saved
syscall frame on the EL1 stack is stale, so the shim drops the frame
and `ERET`s without restoring GPRs. Three call sites use it:

- `sys_execve` (`src/syscall/exec.c:785, 1093`) after the ELF reload.
- `signal_rt_sigreturn` (`src/syscall/signal.c:1710`) after restoring
the saved sigframe.
- `signal_deliver` (`src/syscall/signal.c:1594`) when a signal is
delivered on the syscall-return path; without the marker the shim
would overwrite the handler PC, SP, LR, and arg-register state with
the stale syscall frame on `ERET`.

`X8` (the syscall-number register) and `X9`/`X10` are already considered
clobbered by the Linux syscall ABI, so callers never expect them to be
preserved across SVC.

Important: the first two paths (`sys_execve` and
`signal_rt_sigreturn`) return `SYSCALL_EXEC_HAPPENED` to bypass the
normal syscall dispatch epilogue. `signal_deliver` runs from inside
the epilogue. Any future code path that rebuilds EL0 register state
on the syscall-return path must write `X8 = 2` the same way.
and `ERET`s without restoring GPRs. Two call sites write it, both in
`src/syscall/signal.c`:

- `signal_rt_sigreturn`, after restoring the saved sigframe.
- `deliver_signal_locked`, when a signal is delivered on the
syscall-return path; without the marker the shim would overwrite the
handler PC, SP, LR, and arg-register state with the stale syscall
frame on `ERET`.

`sys_execve` writes no marker: it re-enters through the shim's MMU-off
`_start`, which pops no frame. The `HVC #5` epilogue and both `HVC #9`
W^X tails branch to `exec_drop_frame` on the marker; `handle_brk`
(`HVC #10`) drops its frame on every return.

Linux preserves `X1`-`X30` across `SVC #0`, so the marker must not reach
EL0: a resumed `SVC` that has not executed yet would run as syscall 2
(sysprog21/elfuse#379). Those tails therefore reload `X8` from the
frame's `X8` slot before the pop. The slot holds the `X8` the exception
was taken with unless the host published another there:
`signal_rt_sigreturn` publishes the one it restored, and the `BRK`
ptrace stop the one its tracer left. `signal_rt_sigreturn` also parks
that value for a signal delivered later in the same epilogue, which
would otherwise snapshot the marker as the guest's `X8`; the run loop
drops the record before every `hv_vcpu_run()`, and an inline ptrace
stop that moves the PC re-keys it. The TLBI kinds on the ordinary
syscall-return tail are not covered: a signal delivered there still
records the wire values as `X8`-`X11` (sysprog21/elfuse#384).

Important: `signal_rt_sigreturn` returns `SYSCALL_EXEC_HAPPENED` to
bypass the normal syscall dispatch epilogue, as `sys_execve` does.
`deliver_signal_locked` runs from inside the epilogue. Any future code
path that rebuilds EL0 register state on the syscall-return path must
write `X8 = 2` the same way, and publish the guest's `X8` if it differs
from the one the frame was entered with.

## EL1 Shim And HVC Protocol

Expand All @@ -348,10 +362,10 @@ aligned address from the `Rt` register); HVF traps DC ZVA via `HCR_EL2.TDZ=1`.
| #0 | Normal exit | `X0` = exit code |
| #2 | Bad exception | `X0`=ESR, `X1`=FAR, `X2`=ELR, `X3`=SPSR, `X5`=vector |
| #4 | Set boot system register | `X0` = reg ID (0–8), `X1` = value (used by the shim during boot to install RES1 bits and enable the MMU) |
| #5 | Syscall forward | `X0`–`X5` = args, `X8` = syscall number on entry; on return `X8` carries the TLBI kind (`0` = none, `1` = broadcast, `3` = selective range with `X9` = VA + `X10` = page count, `4` = single-shot `TLBI RVAE1IS` with encoded operand in `X9`). `X8 = 2` is the generic drop-saved-frame marker -- set when the host has rebuilt EL0 state directly (by `execve`, `rt_sigreturn`, and `signal_deliver()` on the syscall-return path) so the shim discards the saved syscall frame on ERET. `X11` is the icache-flush hint (set to `1` when the request transitions a page to executable, so the shim issues `IC` alongside the chosen TLBI) |
| #5 | Syscall forward | `X0`–`X5` = args, `X8` = syscall number on entry; on return `X8` carries the TLBI kind (`0` = none, `1` = broadcast, `3` = selective range with `X9` = VA + `X10` = page count, `4` = single-shot `TLBI RVAE1IS` with encoded operand in `X9`). `X8 = 2` is the generic drop-saved-frame marker -- set when the host has rebuilt EL0 state directly (by `rt_sigreturn` and by signal delivery on the syscall-return path) so the shim discards the saved syscall frame on ERET, reloading only `X8` from it. `X11` is the icache-flush hint (set to `1` when the request transitions a page to executable, so the shim issues `IC` alongside the chosen TLBI) |
| #6 | Embedder extension | `X8` = call number, `X0`–`X7` = args; routed to `g->hvc6_handler` if set, no-op otherwise. Handler may request a vCPU yield via `proc_request_hvc6_yield()` |
| #7 | MRS trap (read sysreg) | host reads register from ESR ISS; returns value in `X0` |
| #9 | W^X toggle | `X0` = FAR, `X1` = type (0 = exec→RX, 1 = write→RW) |
| #9 | W^X toggle | `X0` = FAR, `X1` = type (0 = exec→RX, 1 = write→RW); on return `X8 = 2` when the host answered with a `SIGSEGV` delivery instead of a flip |
| #10 | BRK from EL0 | SIGTRAP delivery / ptrace-stop; GPRs in frame |
| #11 | EL0 fault | SIGSEGV/SIGILL delivery; GPRs in frame |
| #12 | EL0 system-instruction trap | cache maintenance logging (DC CVAU, IC IVAU, …) and `MSR TPIDR_EL0` emulation |
Expand Down Expand Up @@ -797,7 +811,8 @@ In `src/syscall/proc.c`:
`hv_vcpus_exit()`. A stop taken on a syscall return whose tail restores the
saved SVC frame goes through HVC #13, so ptrace snapshots the architectural
GPR set rather than shim scratch. The tails that rebuild EL0 state instead
(`X8 = 2`) already hold that set live, so the host stops on them directly;
(`X8 = 2`) already hold that set live, bar the `X8` the shim reloads from
the saved frame, so the host stops on them directly;
an `execve` re-entry leaves the stop owed for the new image's first
syscall.
- `PTRACE_GETREGSET` / `PTRACE_SETREGSET` (`NT_PRSTATUS`) -- read or write
Expand Down
Loading
Loading