Skip to content

Three audited kernel races recorded, traced and unexecuted: an IPC rollback's expect, chdir against open, an shm map against the last close - #754

Closed
Japabu wants to merge 4 commits into
mainfrom
wt/toyos-audit-kernel
Closed

Japabu wants to merge 4 commits into
mainfrom
wt/toyos-audit-kernel

Conversation

@Japabu

@Japabu Japabu commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Three races an outside audit argued from reading the kernel at b432ed21c, each traced again in the tree and executed by nobody. Nothing is fixed. The diff against main is three files under issues/.

Head e34a45e99, on main 017451624.

Verdicts

REPRODUCED would mean executed; none is. Each is filed kind: defect with one paragraph, "Traced, not executed.", that says what ran, and each names an owner.

T001-SYS-02, IPC setup rollback: TRACED, NOT EXECUTED. sys_inbox_setup (kernel/src/syscall/ipc.rs:466) and connect_through (:265), which sys_namespace_open calls, close a handle with expect after dropping the lock it was installed under. issues/an-ipc-setup-rollback-panics-when-a-sibling-closed-its-new-handle.md

T001-CON-01, chdir against open: TRACED, NOT EXECUTED. kernel/src/syscall/fs.rs:84 keeps its scrutinee's VFS guard across with_process_data; :35 with kernel/src/object/ops.rs takes the two the other way. issues/chdir-holds-the-vfs-lock-into-the-process-data-lock-open-takes-the-other-way.md

T001-MEM-01, shm map against the last close: TRACED, NOT EXECUTED. kernel/src/syscall/ipc.rs:361 clones the object and drops the lock; kernel/src/object/shm.rs:132 maps with no retirement check, :172 tears down once, :186 asserts. issues/an-shm-map-racing-the-last-close-publishes-a-mapping-after-teardown.md

None is already recorded: issues/deferred-release-outlives-its-syscall.md is a late release that loses nothing, and issues/a-user-copy-demand-pages-under-whatever-its-caller-holds.md is another pair of locks.

The fourth finding is not here

The audit's T001-SYS-01, a thread entry SYS_THREAD_SPAWN never checks, was recorded by this branch through f34839395 and is deleted at this head. Draft #757 (wt/toyos-threadentry) carries that issue file in its first commit, 3f4112a88, and deletes it with the refusal the review of this branch asked its exit to be. Its record, its test and its measurement are #757's. The first comment here still holds the test as a patch, and its run on 0c9ff75e7 exited 1 under TCG, the spawn accepted and the process ended alone.

Why three were not executed

No model reaches them: kernel/loom/src/lib.rs takes some twenty kernel files by #[path], and none is the handle table, the object layer or these syscalls; kernel/pure/proclife/interleave.rs models the process table. A guest test needs each window held open, and no SYS_DEBUG action does that for these three paths; a new one is kernel source, outside this branch's fence. Each issue names the actuator it needs. No test that loops for a race was written.

Gates

  • cargo run -- --ci host at e34a45e99: exit 0, [ci] Host: 77 step(s), all green.
  • No guest test is reached by the head's diff, which is issues/ alone.

Unsure

  • The three races are filed kind: defect on a reading nobody has executed.
  • The owners are sites or a track, not people: the IPC rollback's is issues/a-panic-is-never-an-accident.md, whose subject its expect is; the other two name the code that owns the window.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RvnWQFcMuGqTHYhvSnTe8A

Re-traced each against the current tree and recorded it as an issues/ file
with an exit a test can fail; no fix (each fix is its own later change).

- SYS-01: sys_thread_spawn does not bound the thread entry, so a noncanonical
  entry faults the Ring 0 iretq and halt_all_cpus takes the machine.
- SYS-02: SYS_INBOX_SETUP and SYS_NAMESPACE_OPEN install a handle, drop the
  process-data lock, and on a failed fallible step roll back with
  ops::close(...).expect(...); a sibling close makes that expect panic.
- CON-01: sys_chdir holds the match-scrutinee VFS guard into with_process_data
  (VFS then process-data), the inverse of sys_open (process-data then VFS):
  a two-thread deadlock.
- MEM-01: sys_shm_map clones the Arc and maps with no retirement check while
  on_zero_handles tears down exactly once, publishing a mapping after the only
  teardown.

A red test cannot be committed: a clean-tree red breaks --ci host and the merge
queue. The red tests are patches in the pull request; the three races need a new
kernel actuator to stage, a separate change past this branch's fence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvnWQFcMuGqTHYhvSnTe8A
@Japabu

Japabu commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

The SYS-01 test, as run and reversed

Applied to 0c9ff75e7 for the run in the body, then reversed. The machine-test registration exists because a discovered binary runs only on the T14's shared boots; RUST_SKIP keeps it off them.

diff --git a/tests/toyos-rust-tests/src/bin/spawn_noncanonical_entry.rs b/tests/toyos-rust-tests/src/bin/spawn_noncanonical_entry.rs
new file mode 100644
index 000000000..9bbb770b4
--- /dev/null
+++ b/tests/toyos-rust-tests/src/bin/spawn_noncanonical_entry.rs
@@ -0,0 +1,34 @@
+//! A thread entry that is no canonical address is refused, or ends its own
+//! process at most: the machine outlives it.
+
+use toyos_abi::syscall::{self, MmapFlags, MmapProt, SyscallError};
+
+const PAGE_2M: usize = 2 * 1024 * 1024;
+
+/// Canonical under neither 48-bit nor 57-bit addressing.
+const NONCANONICAL: u64 = 0x0100_0000_0000_0000;
+
+fn main() {
+    // SAFETY: a fresh anonymous mapping nothing else names.
+    let stack = unsafe {
+        syscall::mmap(
+            core::ptr::null_mut(),
+            PAGE_2M,
+            MmapProt::READ | MmapProt::WRITE,
+            MmapFlags::ANONYMOUS | MmapFlags::PRIVATE,
+        )
+    };
+    assert!(!stack.is_null(), "no memory for the thread's stack");
+    let base = stack as u64;
+    // SAFETY: the stack is the mapping above; the entry is the input under test.
+    let answer = unsafe { syscall::thread_spawn(NONCANONICAL, base + PAGE_2M as u64, 0, base) };
+    if SyscallError::from_u64(answer).is_none() {
+        // Accepted: the thread's first return to Ring 3 is where the harm is, so wait it out before judging.
+        syscall::thread_join(answer);
+    }
+    assert_eq!(
+        SyscallError::from_u64(answer),
+        Some(SyscallError::InvalidArgument),
+        "a thread entry that is no canonical address was not refused",
+    );
+}
diff --git a/tests/toyos.rs b/tests/toyos.rs
index 996ed86de..555db546f 100644
--- a/tests/toyos.rs
+++ b/tests/toyos.rs
@@ -68,6 +68,9 @@ const ACTUATOR_KERNEL: &[&str] = toyos_build::build::TEST_KERNEL;
 
 // Rust helper binaries that are spawned by tests, not tests themselves.
 const RUST_SKIP: &[&str] = &[
+    // It asks for a thread at an entry no CPU can return to: `thread_entry_noncanonical`
+    // runs it on a boot of its own, since a kernel that takes the entry takes the machine.
+    "spawn_noncanonical_entry",
     // **Its exit code is a measurement, not a verdict**, and the shared block
     // judges every member on `exit=0` alone — so it would red on every boot
     // that measured anything. It also needs the real-time band, which only
@@ -245,6 +248,9 @@ const SCREEN_TESTS: &[(&str, qemu::Profile)] = &[
 /// The tests whose machine shape *is* the test, each on a boot of its own.
 /// `run_machine_test` dispatches them.
 const MACHINE_TESTS: &[&str] = &[
+    // The return to a thread's entry is the CPU's own instruction: no host test
+    // executes it, and the T14's rows share their boot with what it would take down.
+    "thread_entry_noncanonical",
     // Whether QEMU's virtio functions negotiated `VIRTIO_F_ACCESS_PLATFORM`
     // behind its emulated VT-d unit and without one: no shipped machine has a
     // virtio function, so only a QEMU machine can be asked.
@@ -2711,6 +2717,21 @@ fn run_machine_test(name: &str, test_config: &Path) -> Result<(), String> {
         "iommu_virtio_platform" => common::iommu::iommu_virtio_platform(test_config),
         "nested_nmi_is_loud" => faults::nested_nmi_is_loud(test_config),
         "machine_shutdown" => power::machine_shutdown(test_config),
+        "thread_entry_noncanonical" => {
+            let name = "spawn_noncanonical_entry";
+            let tests = compile::repo_root().join("tests/toyos-rust-tests");
+            let bin = (name.to_string(), qemu::build_toyos_bin(qemu::SUITE_ARCH, &tests, name));
+            let mut qemu = QemuInstance::boot_with_options(test_config, &[], &[bin], BootOptions::default());
+            let result = qemu.run_test(&format!("test_rs_{name}"), Duration::from_secs(60));
+            match (result.exit_code, &result.error) {
+                (Some(0), None) => Ok(()),
+                (code, error) => Err(format!(
+                    "exit {code:?}: {}\n{}",
+                    error.as_ref().map_or(String::new(), ToString::to_string),
+                    result.stdout
+                )),
+            }
+        }
         "acpi_power_button" => power::acpi_power_button(test_config),
         "machine_shutdown_short_stop" => power::machine_shutdown_short_stop(test_config),
         other => Err(format!("unknown machine test {other}")),

The three races

No test is written for these: each needs an actuator the kernel does not have, named in its issue, and a test against a hook that does not exist is not a patch anyone can run.

…s under TCG, and three races are traced only

The one sequential finding was run as a guest test under QEMU TCG. The spawn
is accepted and the thread's fault arrives with a Ring 3 frame (cs=0x23), so
the kernel ends that process alone and the machine lives: the halt the audit
claims did not happen on this instrument. Intel's SDM puts the fault in IRET
itself, in Ring 0, and only the T14 can be asked; the issue is renamed to
what was measured, becomes a finding, and its exit is that metal row.

The three interleavings say "traced, not executed" and name the actuator
each needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvnWQFcMuGqTHYhvSnTe8A
@Japabu Japabu changed the title Record four kernel trust-boundary defects an outside audit traced Four audited kernel findings recorded: a noncanonical thread entry ends its process under TCG, three races traced and unexecuted Oct 8, 2026
@Japabu

Japabu commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Review, round 1, of f34839395 against .claude/agents/reviewer.md

Net: +163 lines in 4 files, all under issues/; production 0, tests 0. The branch stands on b432ed21c; kernel/ is byte-identical from there to origin/main 017451624, so every cited line reads the same at both.

What was checked

  • Thread entry (executed, not seen). sys01-run.log carries what the issue quotes and no more: the spawn accepted, FAULT rip=0x0100000000000000, cs=0x0023, pid 10 ended with code=-1, the harness finished, exit 1. The issue claims no halt anyone observed: its hardware paragraph is conditional on Intel's document and says nothing executed it. In the tree: sys_thread_spawn (kernel/src/syscall/proc.rs:132) checks stack_base > stack_ptr only, spawn_thread adds no check of entry, thread_start (kernel/src/arch/x86_64/entry.rs:150) pushes it as the frame's RIP, and fatal_exception (idt/exceptions.rs:483) ends a process only for a Ring 3 frame.
  • IPC rollback (traced). Every step holds: the install at ipc.rs:252 and :455 is under with_process_data and the lock is gone before connector.push and ctx.copy_out; ops::close is HandleTable::remove, which answers Stale or BadHandle for a slot a sibling closed or sent; both rollbacks expect it (:265, :466). A Connection and an Inbox carry TRANSFER, so the send arm holds too.
  • chdir against open (traced). Every step holds: the kernel crate is edition 2021, the VfsGuard is a scrutinee temporary of the match at fs.rs:86 and lives through the Ok arm's with_process_data (process.rs:853); sys_open (fs.rs:35) calls ops::open inside with_process_data, and ops::open takes vfs::lock() at object/ops.rs:95. Both are sync::Lock, whose spin panics at 500M (sync.rs:173).
  • shm map against the last close (traced). Every step holds: HandleTable::get clones the Arc and the lock drops (ipc.rs:362); map_into (shm.rs:132) reads no retirement; HandleEntry::drop retires at zero handles whatever Arcs remain and queues the hook, which runs once (shm.rs:172); a push after it is never taken down, and Drop asserts (shm.rs:186) under [profile.toyos]'s debug-assertions = true.
  • Fields. Four status: open, opened: 2026-10-08; slugs unique; both cited issues exist; no existing issue records any of the four. No machine is named beyond what the tree already names, and no issue carries code.

BLOCKER

  • PR body, "Gates" — no cargo run -- --ci host at f34839395; the body says it was left out because the diff is issues/ alone — Evidence asks it of every branch and a change that touches no source is held to every rule; the host gates read the tree these files join (src/sourcegate.rs). Run it at the head and put the command, its exit code and its closing line in the body.
  • issues/a-thread-entry-is-unchecked-and-a-noncanonical-one-is-measured-only-under-tcg.md:3 — kind: finding — against issues/README.md: a finding is "noticed in passing", and what it is promoted to is "something real that someone should act on — a fix, a measurement, an instrument", with "when unsure, promote". This file owes a measurement by its own exit, and what was executed is already a refusal the kernel did not make: the test that asks for one is red. Change line 3 to kind: defect; make the exit that SYS_THREAD_SPAWN refuses an entry outside the canonical user half, by name, with the first comment's test green; keep the "Not measured on hardware" paragraph and the metal row as the measurement of what an Intel CPU does; delete the fold arm (lines 50-52), since one T14 reading cannot make "the entry is the CPU's to refuse" true of every x86-64 CPU. With that both pull requests agree: an unrun or unseen consequence is a defect carrying one line that says what ran. Three audit claims traced: a BAR asked for twice panics the kernel, netstack keeps a dead client's socket, and a PCI call's slot is not its claim #755's defect marked "Read from the code, not run" is right as filed and changes nothing; this branch's three races are defect already.

NOTE

SEND BACK

…-entry record leaves for the branch that fixes it

- The thread-entry record is deleted here. `wt/toyos-threadentry` carries it
  in its first commit and deletes it with the refusal it asks for, so this
  branch records the three races and nothing else.
- Each of the three names an owner.
- The IPC record gives the install, the push and the rollback to
  `connect_through`, which `sys_namespace_open` calls.
- "Traced, not executed." is a paragraph of its own before each exit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvnWQFcMuGqTHYhvSnTe8A
@Japabu Japabu changed the title Four audited kernel findings recorded: a noncanonical thread entry ends its process under TCG, three races traced and unexecuted Three audited kernel races recorded, traced and unexecuted: an IPC rollback's expect, chdir against open, an shm map against the last close Oct 8, 2026
@Japabu

Japabu commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Answer to review round 1, at e34a45e99

origin/main 017451624 is merged in (28c89a001).

BLOCKER, no --ci host: fixed. cargo run -- --ci host at e34a45e99: exit 0, [ci] Host: 77 step(s), all green. It is in the body.

BLOCKER, the thread-entry issue is a finding: answered by scope, on the orchestrator's instruction. This branch no longer carries that file: draft #757 (wt/toyos-threadentry) carries it in its first commit, 3f4112a88, and deletes it with the refusal this finding asked the exit to be. The kind, the exit and the fold arm are #757's to be judged on.

NOTE, owners: fixed. Each of the three names one: issues/a-panic-is-never-an-accident.md for the IPC rollback's expect, kernel/src/object/shm.rs and sys_chdir in kernel/src/syscall/fs.rs for the other two.

NOTE, the test as a patch nobody can resolve: gone with the file. The test is not committed here; it is #757's.

NOTE, connect_through: fixed. The record gives the install, the push and the rollback to connect_through (ipc.rs:234), which sys_namespace_open (:219) calls.

NOTE, "Traced, not executed.": fixed. A paragraph of its own before the exit in each of the three.

NOTE, the loom sentence: fixed. The body says kernel/loom/src/lib.rs takes some twenty kernel files by #[path], none of them the handle table, the object layer or these syscalls.

@Japabu

Japabu commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Closed without landing: each of the three races it records is now carried, with its issue file, by the pull request that fixes it and deletes it: #758 (the IPC setup rollback), #759 (chdir's lock order) and #762 (the shm map against the last close). The thread-entry issue went to #757 the same way. The review's reading of the three traces stands as the record that each held on main.

@Japabu Japabu closed this Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant