issues: C and C++ conformance is upstream's own suites, judged by their own oracles

[Japabu]

Adds one track issue, issues/build/c-and-cxx-conformance-is-upstreams-own-suites.md. It proposes how ToyOS tests C and C++ conformance once tests/testcases/tinycc/ is replaced. No code moves, and no suite is vendored.

Decisions

• Where the source lives. libc-test and the SingleSource slice are carried in-tree and unmodified under tests/conformance/<suite>/upstream/:
  • The pin is a tier = "suite" row in forks.toml.
  • A suite.toml lock records upstream's git object ids. A host gate compares it with git ls-tree, and --check-forks compares it with upstream.
  • A build-time fetch is ruled out, because tests may not fetch (NOTICE, gbae).
  • A ToyOSOrg mirror repo is ruled out, because it adds a checkout to every worktree and CI job for 11.5 MB.
  • libc++'s tests come from the llvm-project checkout the library is built from, so nothing new is carried.
• Folder structure. tests/conformance/ holds the toyos-conformance crate, ours/, and one directory per suite. Upstream tests are judged only by upstream's oracle.
• One known-red mechanism. src/redlist.rs loads each suite's red rows from its suite.toml, and each row names the stage its test stops at. This replaces NOT_RUN. unsupported is not a red. The pass count is derived on every run and never stored.
• The runner. One std crate, host-tested, that also builds for ToyOS. Its guest binary judges each test and prints one verdict line, which replaces ccheck and the rule held equal in two places. The lit subset the runner implements is the one measured in libc++'s tree. libc++ features are read from __config_site, not declared by hand.
• Tiers. Fast: ours, libc-test, and SingleSource C UnitTests/Regression. Nightly: SingleSource benchmarks, and C++ plus libc++ at c++26 once M3 lands. Weekly: libc++ at c++17.
• Migration. Six steps, each with an exit and a red check. It ends with the 256 TinyCC-derived files, tests/testcases/LICENSE, NOTICE's TinyCC section and CORPUS_POPULATIONS deleted.
• Architectures. Test names carry no architecture. One-arch material says so in suite.toml, and AArch64 conformance stays Tier::Local until the port's stages 7 and 8.

Measured, not estimated

These come from scratch clones and from the primary's llvm-project checkout (read only):

• libc-test 7b95dfa5: 815 files and 1,378,866 bytes are carried. Against sysroot c94bbaad2f72a448, 34 of 275 runnable cases build: 132 stop on a missing fenv.h, 103 on other compile errors, and 6 at the link. The 34 took 2.17 s to compile and link serially, and average 575,217 bytes stripped.
• llvm-test-suite llvmorg-22.1.8: 894 of 4,008 files and 10,175,372 bytes are carried, with 372 reference-output tests.
• libc++ libcxx/test: tree eb1b3645 at the base, at the fork branch and at rust/'s gitlink. The kind and directive counts are in the issue.
• TinyCC 64552b3f: 239 identical files, 17 modified, 60 ours. This matches tests/testcases/LICENSE.

Unsure

• The two licence calls. libc-test's ucb/ vectors are excluded as the same class as the DECUS notice (they forbid redistribution for a fee). Polybench is excluded for its indemnity clause. Both are the owner's to overrule.
• libc++ compile cost is unmeasured, because no ToyOS libc++ exists yet. The 2.5–5 core-hour figure is an estimate.
• The track is longer than issues/README.md's one-screen bar. The brief asked for decisions with their reasons, and I cut everything else.

Gates

• cargo run -- --ci host: EXIT=0 on the committed head 17a2556c.

🤖 Generated with Claude Code

[Japabu]

Review of 17a2556c, round 1. CI: host is success on run 36539479372 at this head. The branch adds prose only (+232 −0, all in issues/build/c-and-cxx-conformance-is-upstreams-own-suites.md; 0 production, 0 test lines), so no test or hardware reading is owed.

Paths below are relative to issues/build/c-and-cxx-conformance-is-upstreams-own-suites.md unless another path is named.

BLOCKER

• :1-233 — The track is a 232-line design: a Decisions section, a folder-structure design, a tier table, a six-step migration table, and a rationale for each choice. issues/README.md:83-89 says a track "does not carry a design, a stage table, a rationale", and that one "grown past a screen is a plan again, and is cut back". The PR body admits this. Cut it to one screen: what is built (three pinned suites plus ours, each judged by its own oracle, and a Rust runner that also builds for ToyOS); what blocks it (34 of 275 libc-test cases build today; M3 for libc++); the constraints (pins, exclusions with their reasons, no fetch, no Python); and the exit (tinycc/, tests/testcases/LICENSE and NOTICE's TinyCC section deleted). Everything else is written as code when it lands.
• :69-76,104-105 — The runner is a host-tested std crate, but it is placed under tests/, which the root Cargo.toml excludes wholesale as "guest programs src/build.rs builds". src/hostws.rs:3-4 requires every host-tested crate to be a workspace member. The tree already has a pattern for this: a top-level member that the guest also builds (toyos-fat32, consumed by tests/toyos-rust-tests). The lit, manifest and oracle library belongs there as a top-level toyos-conformance member. The guest judge belongs in tests/toyos-rust-tests/src/bin/ in place of ccheck.rs. Only the suite data belongs under tests/conformance/.
• :87-102 — "src/redlist.rs stays the one known-red mechanism: one row shape" is false as designed. Under the design, rows move out of DISABLED into N suite.toml files that redlist.rs must parse. The rows gain a stage field. Staged rows run, which contradicts src/redlist.rs:4-5 ("A disabled test runs nowhere"). A third list, unsupported, carries no issue. Either the redlist is redesigned in the open (its header, its row shape and --known-red), or the suite rows live in DISABLED. Decide it against the track's own measurement: libc-test alone needs 241 rows on day one.
• :136 — .gen.py tests are made "red rows against their issues", but they need Python to generate their cases, and Python is ruled out for good. A red whose issue can never close is permanent debt dressed as a defect. Declare them unsupported by that rule, or leave them unregistered.
• :140-143 — The claim "Features are not declared by hand" is contradicted twice. The dialect and target features are "the runner's own", which means hand-declared. The configuration features use "libcxx_macros.py's table", which is either a Rust transcription of upstream's Python table that drifts silently at every pin move, or a parse of a Python file. Name which. If it is a copy, name the gate that reds when upstream's table changes under a new pin. Otherwise this is the second copy held equal that the track says it removes from ccheck.

NOTE

• :41-43 — tier = "suite" is a sibling of forks.toml's existing source tier, which is also pinned, unmodified third-party source. Widen source to cover carried source as well as fetched source instead of adding a tier.
• :47-48 — A lock gate built on git ls-tree HEAD sees only committed trees and needs git, which cannot run inside ToyOS. Git object ids are the SHA-1 of a header plus the bytes, so computing them in Rust over the working tree gives a gate that runs anywhere, including on ToyOS.
• :51-56 — suite.toml holds a copy of SingleSource's CMake knobs, and a gate detects each knob in order to demand that copy. A gate that can find a knob can read its value too. Either say why a copy is needed (conditionals, for example) or have the runner read the knobs directly.
• :62-63 — Adding 39.7 MB of libc++ test source to the toolchain release is a new release payload, decided in passing. Name the job that has no llvm-project checkout and still needs the tests, or compile the tests where the checkout is.
• :156 — The weekly -std=c++17 libc++ slice doubles the most expensive suite, whose cost is only estimated (2.5–5 core-hours). Drop it until a c++17-only failure shows it is needed.
• :97-98 — unsupported carries no issue. It needs a gate that reds when the rule it cites does not exist, or it becomes a silent exclusion list.
• :136 — .sh.cpp red rows need an issue that can close: say what closes them (a shell and lit's substitutions inside the runner) or declare them unsupported.
• :197-199 — Step 5 gates deletion of the LGPL corpus on libc symbol coverage by adopted tests. With 34 of 275 libc-test cases building, that ties the deletion to libc growth for an unbounded time (see owner question 4).

REMOVE

• :11-12 — "Every number here comes from a command that was run": no command is recorded, so a reader cannot check it.
• :18-20,53-55,62-63,112-114,120-126,158-170 — Byte counts, directive counts, CMake knob counts and per-case milliseconds. They go stale at the next pin move and are not constraints.
• :36-39 — The dismissal of the mirror repository is rationale, and "every worktree and every CI job must make" it is unmeasured: worktrees already skip rust/src/llvm-project.
• :215-217 — The claim that the lock is the ledger issues/build/the-third-party-corpus-is-in-no-machine-read-ledger.md asks for. That issue also owns the assets/-is-a-directory half, which the lock does not touch.
• :228-229 — "Long doubles need nothing": not load-bearing.
• PR body, "Measured, not estimated" — Restates the issue's counts into main's merge record.

OWNER-QUESTION

1. :25-26 — libc-test src/math/ucb/ is excluded as "forbid[s] redistribution for a fee, the same class as the DECUS notice". The clause is not quoted. If it is the HPND-style "permission to use, copy, modify, and distribute … without fee is hereby granted", that is conventionally read as a free grant (HPND is OSI-approved), not a ban on selling. Rule on the verbatim text.
2. :28-29 — Polybench is excluded for its clause making the user indemnify OSU. Include or exclude.
3. :24,28 — GPL test data (crlibm, gcc-c-torture) is excluded, while NOTICE carries the LGPL TinyCC corpus as "data, not linked". Which bar does the tree hold test inputs to? Benchmarks/Misc is also dropped wholesale for "per-file terms, one of them PSF", and PSF is redistributable.
4. :197-204 — Is the TinyCC LGPL removal gated on symbol coverage (deferred behind libc growth), or removed at once, accepting the coverage gap?

SEND BACK

🤖 Generated with Claude Code

https://claude.ai/code/session_01U6SVYFkdvV2t38KzNrESxs

[Japabu]

Closed: its known-red mechanism was src/redlist.rs, which #639 deleted, and its Fast/Nightly/Weekly tiers no longer exist; round 1's review was never answered. If C/C++ conformance is wanted, it returns as a fresh one-screen track whose suites run as metal rows under #660's ladder.