diff --git a/docs/flags.md b/docs/flags.md new file mode 100644 index 0000000..97ae0ff --- /dev/null +++ b/docs/flags.md @@ -0,0 +1,176 @@ +# The flag universe + +How many rustc configurations are there, which combinations does rustc refuse, and how many +builds cover every pair or triple of option values? Pinned compiler: nightly-2026-10-06 +(ea137335b) with mirth's local patches (`rustc-verify5`), x86_64-unknown-linux-gnu. + +## Enumeration + +`rustc/flag-universe.py` reads `compiler/rustc_session/src/options.rs`: + +| | options | enumerable | free-form (left out) | +|---|---|---|---| +| `-C` | 51 | 26 | 25 | +| `-Z` | 235 | 190 | 45 | +| total | 286 | 216 | 70 | + +An option's domain is absence plus: `yes`/`no` for a boolean, present for an option without a +value, the backticked values in its parser's description for an enumerated one, `1` and `16` +for a number. Strings, paths, lists, target features, passes and the like are left out, +except 19 options with hand-picked samples (`SAMPLES` in `flag-universe.py`): the first tables +below were made before these were added, and left out `-Copt-level` (its parser takes a +string), so the walks there ran at opt-level 0. With the samples, 235 options are enumerable, +556 values were tried alone and 508 accepted; the pairs were not tried again. + +Every value was tried alone on a one-function lib with `--emit=metadata`: 499 values, 452 +accepted, 212 options with at least one accepted value. Then every pair of accepted values of +different options, and every value rejected alone against every accepted value of every other +option: 122,682 compilations. + +## Which combinations rustc refuses + +Among values accepted alone, only **4 pairs** are rejected together, and they are one rule: +`-Cembed-bitcode=no` with any `-Clto` other than `no`/`off`. + +The 47 values rejected alone fall into four groups: + +- **Not for this target or environment** (left out): `-Zfixed-x18`, `-Zpacked-stack`, + `-Zreg-struct-return`, `-Zregparm`, `-Zinstrument-mcount=fentry-*`, the hwaddress, + kernel-address, kernel-hwaddress, memtag and shadow-call-stack sanitizers, + `-Cinstrument-coverage` (no `profiler_builtins` in our sysroot), out-of-range numbers + (`-Cdwarf-version=1`, `-Zregparm=16`), removed options (`-Zno-parallel-backend`), and a + description whose backticked words are not values (`-Zcross-crate-inline-threshold=no`). +- **Target modifiers** that differ from the sysroot's: `-Zretpoline`, + `-Zretpoline-external-thunk`, `-Zindirect-branch-cs-prefix`, the memory, thread, dataflow + and safestack sanitizers. Accepted once `-Cunsafe-allow-abi-mismatch` names them, which every + row below passes. +- **Needs another option** (pair probe and by hand): + + | value | needs | + |---|---| + | `-Zsplit-lto-unit=yes` | `-Clto` = yes/on/thin/fat | + | `-Zvirtual-function-elimination=yes` | `-Clto` = yes/on/fat | + | `-Zsanitizer=cfi` | `-Clto` = yes/on/fat (thin is not enough) and `-Ccodegen-units=1` | + | `-Zsanitizer=kcfi` | `-Cpanic=abort` | + | `-Zsanitizer-cfi-{diag,recover,minimal-runtime}`, `-Zsanitizer-cfi-canonical-jump-tables=no` | `-Zsanitizer=cfi` | + | `-Zsanitizer-cfi-minimal-runtime=yes` | also `-Zsanitizer-cfi-recover=yes` or `-Zsanitizer-cfi-diag=yes` | + | `-Zsanitizer-cfi-{generalize-pointers,normalize-integers}` | `-Zsanitizer=cfi` or `kcfi` | + | `-Zsanitizer-kcfi-arity=yes` | `-Zsanitizer=kcfi` | + | `-Cforce-frame-pointers=non-leaf`, `-Cpanic=immediate-abort` | `-Zunstable-options` | + | `-Zdump-dep-graph=yes` | `-Zquery-dep-graph=yes` | + +- **Stops compilation early** (left out of walks, and they made the pair probe report false + "needs"): `-Chelp`, `-Zhelp`, `-Zparse-crate-root-only=yes`, `-Zno-analysis`, + `-Zlink-only`, `-Zimplicit-sysroot-deps=no` (needs `#![no_std]`). + +So the declared constraints are few: 1 pairwise exclusion and 17 implications. They are in +`rustc/flag-model.py`, which writes a [PICT](https://github.com/microsoft/pict) model. + +## Covering array sizes + +Values per option are absence plus the accepted values. "untracked" are options marked +`[UNTRACKED]` (no effect on the incremental hash), "tracked" the rest. + +| subset | options | all combinations | t=2 rows (lower bound) | t=3 rows (lower bound) | +|---|---|---|---|---| +| untracked | 57 | 10^27.4 | 42 (36) | 230 (144) | +| tracked | 149 | 10^74.4 | 152 (126) | 1,509 (1,008) | +| all | 206 | 10^102.0 | 166 (126) | 1,746 (1,134) | + +The lower bound is the product of the largest two (three) domains. Without the constraints +the sizes are about the same (all, t=2: 145; tracked, t=3: 1,275); constraints add rows +because the constrained values can only appear in some rows. + +Every row of the all t=2 table and the tracked t=3 table was compiled (metadata, and full +codegen to an rlib) on the trivial crate: **0 rejected** out of 1,675, with 99–137 options set +per row. The constraint list is complete for these tables. + +### Transitions + +An incremental bug needs a change between two sessions. Modeling each option twice (before +and after, 412 parameters for all options) and covering pairs of those covers every +single-option change `x: v → w` and every "x = v before, y = w after": + +| subset | parameters | t=2 rows | t=3 rows | +|---|---|---|---| +| untracked | 114 | 54 | 343 | +| all | 412 | 224 | | + +Each row is a clean build with A, then a rebuild with B, compared with a clean build with B. + +## Walking transitions on sink + +`rustc/flag-walk.py` takes a table from `flag-model.py --transitions --cargo` and, per row, +builds `fixtures/sink` clean with the A options, rebuilds with the B options, builds clean +with the B options, and compares (metadata, object code, binaries, diagnostics, the +program's output). A difference is checked against up to 12 more clean builds first, and +reported as P5 (nondeterminism) if clean builds differ among themselves. + +A real workspace adds constraints the trivial crate does not show (`CARGO_DROP` and +`CARGO_NEEDS` in `flag-model.py`): `-Clto` is rejected for rlibs and dylibs, Cargo's target +probe fails on values that need another option, there are no sanitizer runtimes here, +`-Cprefer-dynamic` with `-Cpanic=abort` or LTO cannot link, and so on. Single values on sink: +428 of 466 build; no single option, set the same in both sessions, changes a rebuild. + +| walk | rows | compared | findings | +|---|---|---|---| +| all options, pairwise transitions | 207 | 190 | 2 rows: P5, finding 10 | +| untracked options, three-way transitions | 345 | 308 | 37 rows: one ICE, finding 11 | + +Three new compiler bugs came out of minimizing rows: 10 and 11 from rows that differed or +crashed, 9 from rows that would not link: + +- **9**: with `-Cno-prepopulate-passes -Zshare-generics=no -Zthinlto=yes`, a dylib fails to + link ([facts](hunt/no-prepopulate-link.md)). Not incremental. +- **10**: with `-g -Clto=thin` and incremental compilation, ThinLTO's input is in codegen + completion order, so clean builds differ from run to run + ([facts](hunt/thinlto-module-order.md)). +- **11**: a session with `-Zprint-type-sizes` leaves a dependency that makes the next + session after an edit panic ([facts](hunt/print-type-sizes-trimmed-paths.md)). + +Not bugs: `-Csplit-debuginfo=packed|unpacked` objects name `.dwo` files by session (the +walk skips object and binary comparison there); `-Zlint-llvm-ir` aborts on a known LLVM lint +finding ([#59793](https://github.com/rust-lang/rust/issues/59793)). With `-Zthreads=4`, `-Zmir-opt-bisect-limit` +makes metadata differ from run to run: the limit counts pass runs across the session, so which +bodies stay under it depends on thread timing (excluded from the models). Rarely, a clean build +reports an extra empty diagnostic: LLVM's `-Zprint-llvm-passes` listing, written from codegen +threads, splits a `-Ztime-passes-format=json` line on stderr and Cargo reads the JSON half as +a message (both left out of `--cargo` models). + +## Staying at the frontier + +A walk that keeps hitting a known bug finds nothing behind it, and excluding the bug from the +model loses the coverage. So a found bug is patched locally and the walk resumes: + +1. `flag-walk.py --pause-on-finding` stops taking rows at the first finding and writes + `PAUSED` (the row and what it found); rows in flight finish. +2. Minimize (`flag-min.py` for failing rows, delta debugging by hand for differences), + reproduce with plain rustc, write the facts (`docs/hunt/`). +3. Patch `~/mirth-work/rust` (a stopgap in `docs/hunt/*-stopgap.patch`; the reproduction in + `docs/hunt/repro.sh` must change), build stage 1, freeze it as a new toolchain. +4. Rerun with `--rustc --recheck`: the rows with findings run first, then the rest. + Done rows are never repeated. + +`rustc/flag-campaign.sh ` strings walks together this way: it exits 3 when a walk pauses, +reads the compiler from the `/rustc` symlink, and resumes on the next run. With the +stopgaps for findings 9–12 (`rustc-verify6`), the 37 rows of the untracked three-way walk +that hit finding 11 all compare equal, and the reproductions of 9–12 no longer fail. + +## Cost + +`fixtures/sink` builds clean in about 2 seconds (dev profile), so one transition row (clean +with A, rebuild with B, clean with B) is a few seconds, and the pairwise transitions walk over +all options (224 rows) is minutes. With fuzzer edits on each row, as in the flag battery +(120 edits per configuration), it is about 27,000 rebuilds, a few hours at the fuzzer's rate +of about 2 edits a second. Three-way over all options at roughly 1,600 rows and 120 edits +each is about a day. + +## Reproduce + + rustc/flag-universe.py --rustc --source --work --jobs 10 + rustc/flag-model.py all model.txt # or untracked / tracked, --transitions + pict model.txt /o:2 /r:1 > rows.tsv + rustc/flag-rows.py rows.tsv --emit=metadata + rustc/flag-model.py untracked tr.txt --transitions --cargo + pict tr.txt /o:3 /r:1 > tr.tsv + rustc/flag-walk.py --rustc --fixture fixtures/sink --flags --table tr.tsv --work diff --git a/docs/hunt.md b/docs/hunt.md index 0393e65..6dcbd4b 100644 --- a/docs/hunt.md +++ b/docs/hunt.md @@ -32,6 +32,12 @@ with `-Zthreads=8`. `rustc/check.sh wide` runs the ordinary checks. | 6 | reused object code keeps the previous checksum of an edited source file in its debuginfo, and with `-Zembed-source` the previous file; with optimizations, ThinLTO symbol names then differ from a clean build | **looks new**; found by the fuzzer at `-Copt-level=2`; root cause found, since 1.44 (#69718); report drafted and a regression test (`hunt/tests/incr-debuginfo-embedded-source`, failing: no fix) | | 7 | warnings from inline assembly are not shown again when an incremental rebuild reuses the codegen unit | **looks new**; found while checking reused codegen units ([`shadow-mode.md`](shadow-mode.md)); on 1.60.0 through the nightly; report drafted ([draft](hunt/issue-asm-warnings-reused-cgu.md)) and a regression test (`hunt/tests/incr-asm-warning-reused`, failing: no fix) | | 8 | with `-Zmir-opt-level=3`, an incremental rebuild encodes an allocation from inlined `core` MIR twice where a clean build encodes it once | **looks new**; found by the fuzzer at `-Zmir-opt-level=4` and named by the reuse check, reduced to one line; on 1.60.0 through the nightly; report drafted ([draft](hunt/issue-inlined-alloc-identity.md)), no fix | +| 9 | a dylib fails to link (undefined hidden symbols) with `-Cno-prepopulate-passes -Zshare-generics=no` and local ThinLTO | **looks new**, not incremental, unstable flags only; found by the flag transitions walk ([`flags.md`](flags.md)); since 1.78; [facts](hunt/no-prepopulate-link.md), local stopgap: no local ThinLTO with `-Cno-prepopulate-passes` ([patch](hunt/no-prepopulate-thinlto-stopgap.patch)) | +| 10 | with `-g -Clto=thin` and incremental compilation, clean builds give different object files from run to run, and since 1.90 (rust-lld) different binaries | **looks new**; found by the flag transitions walk ([`flags.md`](flags.md)), first as a rebuild differing from a clean build; objects differ since at least 1.60; cause found (ThinLTO input in codegen completion order); [facts](hunt/thinlto-module-order.md), local stopgap: inputs sorted by name ([patch](hunt/thinlto-order-stopgap.patch)) | +| 11 | an incremental rebuild after an edit panics ("`trimmed_def_paths` called, diagnostics were expected but none were emitted") when the previous session had `-Zprint-type-sizes` and the crate has an `async fn` awaiting another | **looks new**; found by the three-way walk over untracked option transitions ([`flags.md`](flags.md)); since 1.79; cause found (an awaited type formatted with trimmed paths inside `layout_of`); [facts](hunt/print-type-sizes-trimmed-paths.md), local stopgap: the field type formatted without trimmed paths ([patch](hunt/print-type-sizes-trimmed-stopgap.patch)) | +| 12 | rustc segfaults in LLVM's DWARF emission with `-g -Crelocation-model=rwpi` on x86_64 when the crate has a writable static | **looks new**; stable flags; found by trying `-Crelocation-model` values on sink ([`flags.md`](flags.md)); since 1.60 (LLVM 14); [facts](hunt/rwpi-debuginfo-segfault.md), local stopgap: `rwpi` and `ropi-rwpi` rejected off ARM ([patch](hunt/rwpi-stopgap.patch)) | +| 13 | LLVM's machine outliner (`-Cllvm-args=-enable-machine-outliner`) segfaults with retpolines at `-Copt-level` 1 and up, and fails in other combinations (`-Zcf-protection` with `-Zpatchable-function-entry`, the large code model) | in LLVM; unstable or raw LLVM flags; found by the flag walk with `flag-min.py`; since at least 1.71; [facts](hunt/llvm-retpoline.md); excluded from the models | +| 14 | `-Ccode-model=large` with retpolines: a dylib cannot link (absolute relocation to the retpoline thunk) | in LLVM; unstable flags; found as 13; [facts](hunt/llvm-retpoline.md); excluded from the models | Findings 1 and 2 are single-threaded: an ordinary `cargo build`, an edit, another `cargo build`, and the metadata differs from a clean build of the edited source. Both come diff --git a/docs/hunt/llvm-retpoline.md b/docs/hunt/llvm-retpoline.md new file mode 100644 index 0000000..a494d59 --- /dev/null +++ b/docs/hunt/llvm-retpoline.md @@ -0,0 +1,52 @@ +# Findings 13 and 14: retpoline in LLVM, with the machine outliner and the large code model + +Facts for reports; the reports themselves are for a person to write (rust-lang/rust's LLM +policy). Both are in LLVM's x86 backend, reached through rustc options; LLVM has its own +tracker. No local patch: these are excluded from the flag models instead (they would need an +LLVM build). + +## 13: SIGSEGV with `-Cllvm-args=-enable-machine-outliner` and retpolines + +**Repro.** + + cat > a.rs <<'RS' + pub fn f(v: &[u32]) -> u32 { v.iter().map(|x| x * 3).sum() } + pub fn g(v: &[u32]) -> u32 { v.iter().map(|x| x * 5).sum::() + f(v) } + pub fn h(b: Box u32>) -> u32 { b(1) + b(2) } + RS + rustc --crate-type lib -Copt-level=1 -Cllvm-args=-enable-machine-outliner \ + -Zretpoline=yes -Zunstable-options -Cunsafe-allow-abi-mismatch=retpoline a.rs + +**Expected.** An rlib. **Actual.** `rustc interrupted by SIGSEGV`, exit 139. + +**Conditions.** Any `-Copt-level` from 1 (also `s`, `z`); without retpolines it builds. Before +`-Zretpoline` existed, `-Ctarget-feature=+retpoline-indirect-calls,+retpoline-indirect-branches` +gives the same crash. + +**Versions.** 1.71.0, 1.87.0 (target features), 1.93.0, 1.98.1, nightly-2026-10-06 +(`-Zretpoline`). + +**The outliner fails in other combinations too** (minimized from walk rows with +`rustc/flag-min.py`, on sink): + +- `-Cllvm-args=-enable-machine-outliner -Copt-level=3 -Zcf-protection=full + -Zpatchable-function-entry=4,2 -Cdebuginfo=none`: SIGSEGV. +- `-Cllvm-args=-enable-machine-outliner -Copt-level=2 -Ccode-model=large` (with a few more + options): `error: symbol '.L6$pb' can not be undefined in a subtraction expression`. + +The models now leave the outliner out entirely (`DROP` in `rustc/flag-model.py`). + +## 14: `-Ccode-model=large` with retpolines cannot link a dylib + +**Repro.** + + echo 'pub fn h(b: &dyn Fn(u32) -> u32) -> u32 { b(1) }' > c.rs + rustc --crate-type dylib -Ccode-model=large -Zretpoline=yes \ + -Zunstable-options -Cunsafe-allow-abi-mismatch=retpoline c.rs + +**Expected.** `libc.so`. **Actual.** `rust-lld: error: relocation R_X86_64_64 cannot be used +against symbol '__llvm_retpoline_r11'; recompile with -fPIC`: the call to the retpoline thunk +is emitted as an absolute address in position-independent code. + +**How mirth found them.** The pairwise walk over all options with the sample values +(`docs/flags.md`), then `rustc/flag-min.py` on the rows that failed. diff --git a/docs/hunt/no-prepopulate-link.md b/docs/hunt/no-prepopulate-link.md new file mode 100644 index 0000000..5ee6b4c --- /dev/null +++ b/docs/hunt/no-prepopulate-link.md @@ -0,0 +1,39 @@ +# Finding 9: a dylib fails to link with -Cno-prepopulate-passes and ThinLTO without shared generics + +Facts for a report; the report itself is for a person to write (rust-lang/rust's LLM policy). + +**Repro.** `docs/hunt/repro.sh`, "no-prepopulate-link": + + echo 'pub fn f(a: &mut u8, b: &mut u8) { core::mem::swap(a, b) }' > a.rs + rustc --edition 2021 --crate-type dylib -Ccodegen-units=16 \ + -Cno-prepopulate-passes -Zshare-generics=no -Zthinlto=yes a.rs + +**Expected.** `liba.so`. + +**Actual.** `rust-lld: error: undefined hidden symbol` for +`::unchecked_add::precondition_check`, `core::ptr::swap_nonoverlapping::`, +`<*const ()>::is_aligned_to` and `core::ub_checks::maybe_is_nonoverlapping::runtime`. + +**Conditions.** All three flags are needed; any number of codegen units from 2. A cdylib +fails the same way. A bin and a staticlib link. `-Copt-level=1` with +`-Cno-prepopulate-passes -Zshare-generics=no` (local ThinLTO is on by default there) fails +too. With `-Clto=thin` it links. Not specific to incremental compilation. + +**Variant.** `-Clink-dead-code=yes` in place of `-Zshare-generics=no` fails too, and then a +binary and a cdylib fail as well (a dylib links): `rustc --crate-type bin +-Ccodegen-units=16 -Clink-dead-code=yes -Cno-prepopulate-passes -Zthinlto=yes` on a `main` +calling `core::mem::swap`. Found by minimizing the link failures of the pairwise walk +(`rustc/flag-min.py`). + +**Versions.** Stable releases with `RUSTC_BOOTSTRAP=1`: links on 1.53.0 through 1.77.0, +fails on 1.78.0 through 1.98.1 and nightly-2026-10-06. 1.78 is when these `ub_checks` +helpers appeared in `core`, so older releases may only lack a function that shows it. + +**Cause, as far as followed.** With `-Csave-temps`, the defining codegen unit's copy is +`define hidden` in `thin-lto-input` and `define internal` from `thin-lto-after-internalize` +on, while the calling unit still only `declare`s it after `thin-lto-after-import`: ThinLTO +internalized a definition another module needs and the import did not happen. Why +`-Cno-prepopulate-passes` changes this was not followed further. + +**How mirth found it.** The transitions walk on `fixtures/sink` (`docs/flags.md`), whose +`sink-dy` crate is a dylib, then a delta minimization of the row's 130 flags. diff --git a/docs/hunt/no-prepopulate-thinlto-stopgap.patch b/docs/hunt/no-prepopulate-thinlto-stopgap.patch new file mode 100644 index 0000000..778b0d9 --- /dev/null +++ b/docs/hunt/no-prepopulate-thinlto-stopgap.patch @@ -0,0 +1,15 @@ +--- a/compiler/rustc_session/src/session.rs ++++ b/compiler/rustc_session/src/session.rs +@@ -1036,6 +1036,12 @@ + return config::Lto::No; + } + ++ // Without the default passes, local ThinLTO internalizes definitions that another ++ // codegen unit still references, and linking fails. ++ if self.opts.cg.no_prepopulate_passes { ++ return config::Lto::No; ++ } ++ + // If `-Z thinlto` specified process that, but note that this is mostly + // a deprecated option now that `-C lto=thin` exists. + if let Some(enabled) = self.opts.unstable_opts.thinlto { diff --git a/docs/hunt/print-type-sizes-trimmed-paths.md b/docs/hunt/print-type-sizes-trimmed-paths.md new file mode 100644 index 0000000..114e835 --- /dev/null +++ b/docs/hunt/print-type-sizes-trimmed-paths.md @@ -0,0 +1,48 @@ +# Finding 11: an incremental rebuild ICEs after a session with -Zprint-type-sizes + +Facts for a report; the report itself is for a person to write (rust-lang/rust's LLM policy). + +**Repro.** `docs/hunt/repro.sh`, "print-type-sizes-ice": + + cat > lib.rs <<'RS' + pub async fn inner() -> u32 { 1 } + pub async fn outer() -> u32 { let s = String::from("x"); inner().await + s.len() as u32 } + pub fn make() -> impl std::future::Future { outer() } + RS + rustc --edition 2021 --crate-type lib -Cincremental=inc -Zprint-type-sizes lib.rs + echo 'pub fn g() {}' >> lib.rs + rustc --edition 2021 --crate-type lib -Cincremental=inc lib.rs + +**Expected.** The second build succeeds, as a clean build of the edited file does. + +**Actual.** `thread 'rustc' panicked at compiler/rustc_errors/src/lib.rs:458:17:` +"`trimmed_def_paths` called, diagnostics were expected but none were emitted". Exit 101. + +**Conditions.** The edit is needed (without it every node stays green and nothing reruns); any +edit that changes HIR does. The second session must not have `-Zprint-type-sizes`, +`-Zquery-dep-graph`, `-Zdump-mir`, `-Zunpretty` or `RUSTC_LOG`, which exempt a session from +the check. Needs a coroutine that awaits another future. + +**Versions.** With `RUSTC_BOOTSTRAP=1`: fine on 1.60.0 through 1.78.0, panics on 1.79.0 +through 1.98.1 and nightly-2026-10-06. + +**Cause.** `variant_info_for_coroutine` in `rustc_ty_utils/src/layout.rs` formats the type of +an `__awaitee` field with `field_layout.ty.to_string()`, which uses trimmed paths, inside the +`layout_of` query (via `record_layout_for_printing`, under `-Zprint-type-sizes`). The type of +the whole layout is formatted under `with_no_trimmed_paths!`, this field type is not. In the +first session `Session::record_trimmed_def_paths` returns early because of +`-Zprint-type-sizes`, and the dependency graph records `layout_of -> trimmed_def_paths`. In +the next session, `try_mark_green` for a body query reaches that edge; `trimmed_def_paths` +has a changed input (the edit) and is executed, and without the exemption it sets +`must_produce_diag`, which panics at the end of the session. Backtrace: +`set_must_produce_diag` ← `record_trimmed_def_paths` ← `trimmed_def_paths` ← +`try_execute_query` ← `try_mark_previous_green` ← `ensure_can_skip_execution` ← +`par_hir_body_owners` (`run_required_analyses`). + +`-Zprint-type-sizes` is `[UNTRACKED]`, so the second session reuses the first's graph. + +**How mirth found it.** The three-way walk over untracked option transitions on +`fixtures/sink` (`docs/flags.md`): 37 of 345 rows, all with `-Zprint-type-sizes=yes` before +and without it after. A delta minimization left `-Zprint-type-sizes` before and +`-Zvalidate-mir` after; on sink no edit is needed; which input changes there was not +followed. diff --git a/docs/hunt/print-type-sizes-trimmed-stopgap.patch b/docs/hunt/print-type-sizes-trimmed-stopgap.patch new file mode 100644 index 0000000..60ac55f --- /dev/null +++ b/docs/hunt/print-type-sizes-trimmed-stopgap.patch @@ -0,0 +1,11 @@ +--- a/compiler/rustc_ty_utils/src/layout.rs ++++ b/compiler/rustc_ty_utils/src/layout.rs +@@ -1050,7 +1050,7 @@ + // Include the type name if there is no field name, or if the name is the + // __awaitee placeholder symbol which means a child future being `.await`ed. + type_name: (field_name.is_none() || field_name == Some(sym::__awaitee)) +- .then(|| Symbol::intern(&field_layout.ty.to_string())), ++ .then(|| Symbol::intern(&with_no_trimmed_paths!(field_layout.ty.to_string()))), + } + }) + .chain(upvar_fields.iter().copied()) diff --git a/docs/hunt/report-untracked.patch b/docs/hunt/report-untracked.patch index da4902b..9e315ec 100644 --- a/docs/hunt/report-untracked.patch +++ b/docs/hunt/report-untracked.patch @@ -1,3 +1,29 @@ +diff --git a/compiler/rustc_ast_lowering/src/index.rs b/compiler/rustc_ast_lowering/src/index.rs +index 3795673d5..30992a65b 100644 +--- a/compiler/rustc_ast_lowering/src/index.rs ++++ b/compiler/rustc_ast_lowering/src/index.rs +@@ -96,7 +96,7 @@ fn insert(&mut self, span: Span, hir_id: HirId, node: Node<'hir>) { + hir_id.owner, + ) + } +- if self.tcx.sess.opts.incremental.is_some() ++ if (*self.tcx.sess.opts.read_incremental()).is_some() + && span.parent().is_none() + && !span.is_dummy() + { +diff --git a/compiler/rustc_ast_lowering/src/lib.rs b/compiler/rustc_ast_lowering/src/lib.rs +index b6d101766..44a30844e 100644 +--- a/compiler/rustc_ast_lowering/src/lib.rs ++++ b/compiler/rustc_ast_lowering/src/lib.rs +@@ -1024,7 +1024,7 @@ fn mark_span_with_reason( + + fn span_lowerer(&self) -> SpanLowerer { + SpanLowerer { +- is_incremental: self.tcx.sess.opts.incremental.is_some(), ++ is_incremental: (*self.tcx.sess.opts.read_incremental()).is_some(), + def_id: self.curr_owner.owner_id().def_id, + } + } diff --git a/compiler/rustc_attr_parsing/src/attributes/rustc_internal.rs b/compiler/rustc_attr_parsing/src/attributes/rustc_internal.rs index b9c03883d..d51e6b650 100644 --- a/compiler/rustc_attr_parsing/src/attributes/rustc_internal.rs @@ -103,8 +129,30 @@ index 1f8177477..3b51986d6 100644 || tcx.sess.opts.unstable_opts.polonius.is_legacy_enabled() } +diff --git a/compiler/rustc_codegen_gcc/src/lib.rs b/compiler/rustc_codegen_gcc/src/lib.rs +index b82d05dc2..4bb017062 100644 +--- a/compiler/rustc_codegen_gcc/src/lib.rs ++++ b/compiler/rustc_codegen_gcc/src/lib.rs +@@ -199,7 +199,7 @@ fn file_paths(sysroot_path: &Path, sess: &EarlySession) -> Vec { + // We use all_paths() instead of only path() in case the path specified by --sysroot is + // invalid. + // This is the case for instance in Rust for Linux where they specify --sysroot=/dev/null. +- 'sysroot: for path in sess.opts.sysroot.all_paths() { ++ 'sysroot: for path in (*sess.opts.read_sysroot()).all_paths() { + for libgccjit_target_lib_file in file_paths(path, sess) { + if let Ok(true) = fs::exists(&libgccjit_target_lib_file) { + load_libgccjit_if_needed(&libgccjit_target_lib_file); +@@ -210,7 +210,7 @@ fn file_paths(sysroot_path: &Path, sess: &EarlySession) -> Vec { + + if !gccjit::is_loaded() { + let mut paths = vec![]; +- for path in sess.opts.sysroot.all_paths() { ++ for path in (*sess.opts.read_sysroot()).all_paths() { + for libgccjit_target_lib_file in file_paths(path, sess) { + paths.push(libgccjit_target_lib_file); + } diff --git a/compiler/rustc_codegen_llvm/src/back/llvm_backend.rs b/compiler/rustc_codegen_llvm/src/back/llvm_backend.rs -index 374c78070..95628cdcd 100644 +index b5d721623..c9aa88259 100644 --- a/compiler/rustc_codegen_llvm/src/back/llvm_backend.rs +++ b/compiler/rustc_codegen_llvm/src/back/llvm_backend.rs @@ -403,7 +403,7 @@ fn join_codegen( @@ -174,9 +222,18 @@ index a582e897e..859a2e5c4 100644 } diff --git a/compiler/rustc_codegen_ssa/src/assert_module_sources.rs b/compiler/rustc_codegen_ssa/src/assert_module_sources.rs -index 431783552..8013ec2bd 100644 +index 431783552..c054a1d2c 100644 --- a/compiler/rustc_codegen_ssa/src/assert_module_sources.rs +++ b/compiler/rustc_codegen_ssa/src/assert_module_sources.rs +@@ -41,7 +41,7 @@ + #[allow(missing_docs)] + pub fn assert_module_sources(tcx: TyCtxt<'_>, set_reuse: &dyn Fn(&mut CguReuseTracker)) { + tcx.dep_graph.with_ignore(|| { +- if tcx.sess.opts.incremental.is_none() { ++ if (*tcx.sess.opts.read_incremental()).is_none() { + return; + } + @@ -55,7 +55,7 @@ pub fn assert_module_sources(tcx: TyCtxt<'_>, set_reuse: &dyn Fn(&mut CguReuseTr let mut ams = AssertModuleSource { tcx, @@ -218,7 +275,7 @@ index 18b5c001f..46e2ccde7 100644 } diff --git a/compiler/rustc_codegen_ssa/src/back/link.rs b/compiler/rustc_codegen_ssa/src/back/link.rs -index 45ab9b8bc..6aaa60461 100644 +index 45ab9b8bc..6e6d0767f 100644 --- a/compiler/rustc_codegen_ssa/src/back/link.rs +++ b/compiler/rustc_codegen_ssa/src/back/link.rs @@ -353,7 +353,7 @@ pub fn link_binary( @@ -239,6 +296,33 @@ index 45ab9b8bc..6aaa60461 100644 return; } +@@ -902,7 +902,7 @@ fn link_staticlib( + + all_native_libs.extend_from_slice(&crate_info.used_libraries); + +- for print in &sess.opts.prints { ++ for print in &(*sess.opts.read_prints()) { + if print.kind == PrintKind::NativeStaticLibs { + print_native_static_libs(sess, &print.out, &all_native_libs, &all_rust_dylibs); + } +@@ -1279,7 +1279,7 @@ fn link_natively( + cmd.env_remove(k.as_ref()); + } + +- for print in &sess.opts.prints { ++ for print in &(*sess.opts.read_prints()) { + if print.kind == PrintKind::LinkArgs { + let content = format!("{cmd:?}\n"); + print.out.overwrite(&content, sess); +@@ -1426,7 +1426,7 @@ fn link_natively( + command: cmd, + escaped_output, + verbose: sess.opts.verbose, +- sysroot_dir: sess.opts.sysroot.path().to_owned(), ++ sysroot_dir: (*sess.opts.read_sysroot()).path().to_owned(), + }; + sess.dcx().emit_err(err); + // If MSVC's `link.exe` was expected but the return code @@ -1535,7 +1535,7 @@ fn link_natively( } } @@ -257,6 +341,15 @@ index 45ab9b8bc..6aaa60461 100644 // Linking against in-tree sanitizer runtimes is disabled via // `-Z external-clangrt` return; +@@ -1796,7 +1796,7 @@ fn find_sanitizer_runtime(sess: &Session, filename: &str) -> PathBuf { + sess.target_tlib_path.dir.to_path_buf() + } else { + filesearch::make_target_lib_path( +- &sess.opts.sysroot.default, ++ &(*sess.opts.read_sysroot()).default, + sess.opts.target_triple.tuple(), + ) + } @@ -1924,11 +1924,11 @@ fn adjust_flavor_to_features( } } @@ -280,6 +373,15 @@ index 45ab9b8bc..6aaa60461 100644 return ret; } +@@ -2299,7 +2299,7 @@ fn detect_self_contained_mingw(sess: &Session, linker: &Path) -> bool { + for dir in env::split_paths(&env::var_os("PATH").unwrap_or_default()) { + let full_path = dir.join(&linker_with_extension); + // If linker comes from sysroot assume self-contained mode +- if full_path.is_file() && !full_path.starts_with(sess.opts.sysroot.path()) { ++ if full_path.is_file() && !full_path.starts_with((*sess.opts.read_sysroot()).path()) { + return false; + } + } @@ -2317,7 +2317,7 @@ fn self_contained_components( // Turn the backwards compatible bool values for `self_contained` into fully inferred // `LinkSelfContainedComponents`. @@ -325,6 +427,15 @@ index 45ab9b8bc..6aaa60461 100644 let libs = crate_info .used_crates .iter() +@@ -3207,7 +3207,7 @@ fn linker_with_args( + + // Only LLD supports controlling parallelism at the moment. + let mut tokens = Vec::new(); +- if let LinkerJobs::Explicit(limit) = sess.opts.jobs.linker ++ if let LinkerJobs::Explicit(limit) = (*sess.opts.read_jobs()).linker + && flavor.uses_lld() + { + // Try obtaining as many jobserver tokens as possible (within the limit) to run parallel @@ -3341,11 +3341,11 @@ fn add_order_independent_options( ); @@ -363,7 +474,7 @@ index 45ab9b8bc..6aaa60461 100644 for path in sess.get_tools_search_paths(false) { let linker_path = path.join("gcc-ld"); diff --git a/compiler/rustc_codegen_ssa/src/back/linker.rs b/compiler/rustc_codegen_ssa/src/back/linker.rs -index ffb9130ed..9e37a7cb7 100644 +index ffb9130ed..6681f1d45 100644 --- a/compiler/rustc_codegen_ssa/src/back/linker.rs +++ b/compiler/rustc_codegen_ssa/src/back/linker.rs @@ -73,7 +73,7 @@ pub(crate) fn get_linker<'a>( @@ -384,8 +495,17 @@ index ffb9130ed..9e37a7cb7 100644 let mut rpath = OsString::from("@rpath/"); rpath.push(out_filename.file_name().unwrap()); self.link_arg("-install_name").link_arg(rpath); +@@ -1095,7 +1095,7 @@ fn debuginfo(&mut self, _strip: Strip, natvis_debugger_visualizers: &[PathBuf]) + self.link_arg("/PDBALTPATH:%_PDB%"); + + // This will cause the Microsoft linker to embed .natvis info into the PDB file +- let natvis_dir_path = self.sess.opts.sysroot.path().join("lib\\rustlib\\etc"); ++ let natvis_dir_path = (*self.sess.opts.read_sysroot()).path().join("lib\\rustlib\\etc"); + if let Ok(natvis_dir) = fs::read_dir(&natvis_dir_path) { + for entry in natvis_dir { + match entry { diff --git a/compiler/rustc_codegen_ssa/src/back/write.rs b/compiler/rustc_codegen_ssa/src/back/write.rs -index 04be7956e..3d6cdf857 100644 +index 04be7956e..7a9c5ee83 100644 --- a/compiler/rustc_codegen_ssa/src/back/write.rs +++ b/compiler/rustc_codegen_ssa/src/back/write.rs @@ -126,7 +126,7 @@ macro_rules! if_regular { @@ -397,6 +517,24 @@ index 04be7956e..3d6cdf857 100644 let should_emit_obj = sess.opts.output_types.contains_key(&OutputType::Exe) || match kind { +@@ -410,7 +410,7 @@ fn need_bitcode_in_object(tcx: TyCtxt<'_>) -> bool { + } + + fn need_pre_lto_bitcode_for_incr_comp(sess: &Session) -> bool { +- if sess.opts.incremental.is_none() { ++ if (*sess.opts.read_incremental()).is_none() { + return false; + } + +@@ -465,7 +465,7 @@ fn copy_all_cgu_workproducts_to_incr_comp_cache_dir( + ) -> WorkProductMap { + let mut work_products = WorkProductMap::default(); + +- if sess.opts.incremental.is_none() || sess.opts.unstable_opts.disable_incr_comp_backend_caching ++ if (*sess.opts.read_incremental()).is_none() || sess.opts.unstable_opts.disable_incr_comp_backend_caching + { + return work_products; + } @@ -536,7 +536,7 @@ pub fn produce_final_output_artifacts( } else { copy_gracefully(&path, &output); @@ -415,6 +553,15 @@ index 04be7956e..3d6cdf857 100644 // Remove the temporary .#module-name#.rcgu.o objects. If the user didn't // explicitly request bitcode (with --emit=bc), and the bitcode is not // needed for building an rlib, then we must remove .#module-name#.bc as +@@ -1226,7 +1226,7 @@ fn start_executing_work( + // Note that using `jobserver::Proxy` is not necessary here, the code below always acquires + // tokens before releasing them, so we can never accidentally release the last token + // permanently held by rustc process. +- let parallel = match sess.opts.jobs.backend { ++ let parallel = match (*sess.opts.read_jobs()).backend { + Some(n) if backend.supports_parallel() => Some(n), + _ => None, + }; @@ -1242,7 +1242,7 @@ fn start_executing_work( let opt_level = tcx.backend_optimization_level(()); let tm_factory = backend.target_machine_factory(tcx.sess, opt_level); @@ -424,7 +571,14 @@ index 04be7956e..3d6cdf857 100644 let result = fs::create_dir_all(dir).and_then(|_| dir.canonicalize()); match result { Ok(dir) => Some(dir), -@@ -1258,12 +1258,12 @@ fn start_executing_work( +@@ -1252,18 +1252,18 @@ fn start_executing_work( + None + }; + +- let stack_size = sess.opts.recommended_stack_size; ++ let stack_size = (*sess.opts.read_recommended_stack_size()); + + let cgcx = CodegenContext { crate_types: tcx.crate_types().to_vec(), lto: sess.lto(), use_linker_plugin_lto: sess.opts.cg.linker_plugin_lto.enabled(), @@ -441,8 +595,17 @@ index 04be7956e..3d6cdf857 100644 remark_dir, old_incr_comp_session_dir: tcx .incr_comp_session +@@ -2144,7 +2144,7 @@ pub fn join( + &crate_info.exported_symbols_for_lto, + &crate_info.each_linked_rlib_file_for_lto, + needs_thin_lto, +- sess.opts.recommended_stack_size, ++ (*sess.opts.read_recommended_stack_size()), + ), + allocator_module: None, + } diff --git a/compiler/rustc_codegen_ssa/src/base.rs b/compiler/rustc_codegen_ssa/src/base.rs -index fea558316..703270e78 100644 +index d6f8447fe..2ffb55c11 100644 --- a/compiler/rustc_codegen_ssa/src/base.rs +++ b/compiler/rustc_codegen_ssa/src/base.rs @@ -809,7 +809,7 @@ pub fn codegen_crate< @@ -514,7 +677,7 @@ index 000000000..1ee27a720 + } +} diff --git a/compiler/rustc_driver_impl/src/lib.rs b/compiler/rustc_driver_impl/src/lib.rs -index 1ba7ce953..3bd2fdb0f 100644 +index 1ba7ce953..69217868c 100644 --- a/compiler/rustc_driver_impl/src/lib.rs +++ b/compiler/rustc_driver_impl/src/lib.rs @@ -162,8 +162,8 @@ fn config(&mut self, config: &mut interface::Config) { @@ -523,11 +686,20 @@ index 1ba7ce953..3bd2fdb0f 100644 // - self.time_passes = (config.opts.prints.is_empty() && config.opts.unstable_opts.time_passes) - .then_some(config.opts.unstable_opts.time_passes_format); -+ self.time_passes = (config.opts.prints.is_empty() && (*config.opts.unstable_opts.read_time_passes())) ++ self.time_passes = ((*config.opts.read_prints()).is_empty() && (*config.opts.unstable_opts.read_time_passes())) + .then_some((*config.opts.unstable_opts.read_time_passes_format())); config.opts.trimmed_def_paths = true; } } +@@ -245,7 +245,7 @@ pub fn compiler_entrypoint(at_args: &[String], callbacks: &mut (dyn Callbacks + + // This implements `-Whelp`. It should be handled very early, like + // `--help`/`-Zhelp`/`-Chelp`. This is the earliest it can run, because + // it must happen after lints are registered, during session creation. +- if sess.opts.describe_lints { ++ if (*sess.opts.read_describe_lints()) { + describe_lints(sess, registered_lints); + return; + } @@ -263,7 +263,7 @@ pub fn compiler_entrypoint(at_args: &[String], callbacks: &mut (dyn Callbacks + sess.dcx().fatal("no input filename given"); // this is fatal } @@ -537,6 +709,15 @@ index 1ba7ce953..3bd2fdb0f 100644 list_metadata(sess, &*codegen_backend.metadata_loader()); return; } +@@ -278,7 +278,7 @@ pub fn compiler_entrypoint(at_args: &[String], callbacks: &mut (dyn Callbacks + + let mut krate = passes::parse(sess); + + // If pretty printing is requested: Figure out the representation, print it and exit +- if let Some(pp_mode) = sess.opts.pretty { ++ if let Some(pp_mode) = (*sess.opts.read_pretty()) { + if pp_mode.needs_ast_map() { + create_and_enter_global_ctxt(compiler, krate, |tcx| { + tcx.ensure_ok().early_lint_checks(()); @@ -296,7 +296,7 @@ pub fn compiler_entrypoint(at_args: &[String], callbacks: &mut (dyn Callbacks + return; } @@ -573,6 +754,33 @@ index 1ba7ce953..3bd2fdb0f 100644 sess.cfg_version, ) { if path.extension().is_some_and(|extension| extension == "rs") { +@@ -639,7 +639,7 @@ pub fn print_crate_info( + + // NativeStaticLibs and LinkArgs are special - printed during linking + // (empty iterator returns true) +- if sess.opts.prints.iter().all(|p| p.kind == NativeStaticLibs || p.kind == LinkArgs) { ++ if (*sess.opts.read_prints()).iter().all(|p| p.kind == NativeStaticLibs || p.kind == LinkArgs) { + return Compilation::Continue; + } + +@@ -656,7 +656,7 @@ pub fn print_crate_info( + None + }; + +- for req in &sess.opts.prints { ++ for req in &(*sess.opts.read_prints()) { + let mut crate_info = String::new(); + macro println_info($($arg:tt)*) { + crate_info.write_fmt(format_args!("{}\n", format_args!($($arg)*))).unwrap() +@@ -670,7 +670,7 @@ pub fn print_crate_info( + } + HostTuple => println_info!("{}", rustc_session::config::host_tuple()), + WasmProcMacroTuple => println_info!("{}", sess.wasm_proc_macro_tuple), +- Sysroot => println_info!("{}", sess.opts.sysroot.path().display()), ++ Sysroot => println_info!("{}", (*sess.opts.read_sysroot()).path().display()), + TargetLibdir => println_info!("{}", sess.target_tlib_path.dir.display()), + TargetSpecJson => { + println_info!("{}", serde_json::to_string_pretty(&sess.target.to_json()).unwrap()); diff --git a/compiler/rustc_expand/src/base.rs b/compiler/rustc_expand/src/base.rs index 01e8cef49..b72b27d7d 100644 --- a/compiler/rustc_expand/src/base.rs @@ -587,9 +795,18 @@ index 01e8cef49..b72b27d7d 100644 let stability = find_attr!(attrs, Stability { stability, .. } => *stability); diff --git a/compiler/rustc_expand/src/expand.rs b/compiler/rustc_expand/src/expand.rs -index 5a2442fbc..3ec74c962 100644 +index 5a2442fbc..11d36c625 100644 --- a/compiler/rustc_expand/src/expand.rs +++ b/compiler/rustc_expand/src/expand.rs +@@ -652,7 +652,7 @@ fn collect_invocations( + .resolver + .visit_ast_fragment_with_placeholders(self.cx.current_expansion.id, &fragment); + +- if self.cx.sess.opts.incremental.is_some() { ++ if (*self.cx.sess.opts.read_incremental()).is_some() { + for (invoc, _) in invocations.iter_mut() { + let expn_id = invoc.expansion_data.id; + let parent_def = self.cx.resolver.invocation_parent(expn_id); @@ -719,7 +719,7 @@ fn expand_invoc( return ExpandResult::Ready(invoc.fragment_kind.dummy(invoc.span(), guar)); } @@ -599,6 +816,19 @@ index 5a2442fbc..3ec74c962 100644 let (fragment_kind, span) = (invoc.fragment_kind, invoc.span()); ExpandResult::Ready(match invoc.kind { +diff --git a/compiler/rustc_expand/src/proc_macro.rs b/compiler/rustc_expand/src/proc_macro.rs +index 5e01b851b..81c84a9a8 100644 +--- a/compiler/rustc_expand/src/proc_macro.rs ++++ b/compiler/rustc_expand/src/proc_macro.rs +@@ -110,7 +110,7 @@ fn expand( + + let invoc_id = ecx.current_expansion.id; + +- let res = if ecx.sess.opts.incremental.is_some() ++ let res = if (*ecx.sess.opts.read_incremental()).is_some() + && ecx.sess.opts.unstable_opts.cache_proc_macros + { + (*EXPAND_DERIVE_MACRO_CACHED)(invoc_id, input, ecx, self.client) diff --git a/compiler/rustc_hir_typeck/src/upvar.rs b/compiler/rustc_hir_typeck/src/upvar.rs index 98155d9a8..e729e0537 100644 --- a/compiler/rustc_hir_typeck/src/upvar.rs @@ -667,10 +897,55 @@ index 853a5c9ba..c6d38410f 100644 eprintln!( "[incremental] ignoring cache artifact `{}`: {}", file.file_name().unwrap().to_string_lossy(), +diff --git a/compiler/rustc_incremental/src/persist/fs.rs b/compiler/rustc_incremental/src/persist/fs.rs +index 3f896584e..bc45e5418 100644 +--- a/compiler/rustc_incremental/src/persist/fs.rs ++++ b/compiler/rustc_incremental/src/persist/fs.rs +@@ -223,7 +223,7 @@ pub(crate) fn prepare_session_directory( + crate_name: Symbol, + stable_crate_id: StableCrateId, + ) -> IncrCompSession { +- assert!(sess.opts.incremental.is_some()); ++ assert!((*sess.opts.read_incremental()).is_some()); + + let _timer = sess.timer("incr_comp_prepare_session_directory"); + +@@ -281,7 +281,7 @@ pub fn finalize_session_directory( + ) { + assert!(sess.dcx().has_errors_or_delayed_bugs().is_none()); + +- if sess.opts.incremental.is_none() { ++ if (*sess.opts.read_incremental()).is_none() { + return; + } + let mut incr_comp_session = incr_comp_session.unwrap(); +@@ -540,7 +540,7 @@ fn string_to_timestamp(s: &str) -> Result { + } + + fn crate_path(sess: &Session, crate_name: Symbol, stable_crate_id: StableCrateId) -> PathBuf { +- let incr_dir = sess.opts.incremental.as_ref().unwrap().clone(); ++ let incr_dir = (*sess.opts.read_incremental()).as_ref().unwrap().clone(); + + let crate_name = + format!("{crate_name}-{}", stable_crate_id.as_u64().to_base_fixed_len(CASE_INSENSITIVE)); diff --git a/compiler/rustc_incremental/src/persist/load.rs b/compiler/rustc_incremental/src/persist/load.rs -index 76e0bf92c..0caeb2789 100644 +index 76e0bf92c..89dd50c21 100644 --- a/compiler/rustc_incremental/src/persist/load.rs +++ b/compiler/rustc_incremental/src/persist/load.rs +@@ -33,11 +33,11 @@ enum LoadResult { + } + + fn load_dep_graph(sess: &Session, incr_comp_session: &IncrCompSession) -> LoadResult { +- assert!(sess.opts.incremental.is_some()); ++ assert!((*sess.opts.read_incremental()).is_some()); + + let _timer = sess.prof.generic_activity("incr_comp_prepare_load_dep_graph"); + +- // Calling `sess.incr_comp_session_dir()` will panic if `sess.opts.incremental.is_none()`. ++ // Calling `sess.incr_comp_session_dir()` will panic if `(*sess.opts.read_incremental()).is_none()`. + // Fortunately, we just checked that this isn't the case. + let Some(path) = old_dep_graph_path(incr_comp_session) else { + return LoadResult::DataOutOfDate; @@ -64,7 +64,7 @@ fn load_dep_graph(sess: &Session, incr_comp_session: &IncrCompSession) -> LoadRe for swp in work_products { let all_files_exist = swp.work_product.saved_files.items().all(|(_, path)| { @@ -689,6 +964,15 @@ index 76e0bf92c..0caeb2789 100644 eprintln!( "[incremental] completely ignoring cache because of \ differing commandline arguments" +@@ -122,7 +122,7 @@ pub fn load_query_result_cache( + sess: &Session, + incr_comp_session: Option<&IncrCompSession>, + ) -> Option { +- if sess.opts.incremental.is_none() { ++ if (*sess.opts.read_incremental()).is_none() { + return None; + } + let incr_comp_session = incr_comp_session.unwrap(); @@ -150,7 +150,7 @@ pub fn load_query_result_cache( /// the outcome of trying to load previous-session state. fn maybe_assert_incr_state(sess: &Session, load_result: &LoadResult) { @@ -698,6 +982,28 @@ index 76e0bf92c..0caeb2789 100644 // Match exhaustively to make sure we don't miss any cases. let loaded = match load_result { +@@ -182,7 +182,7 @@ pub fn setup_dep_graph( + crate_name: Symbol, + stable_crate_id: StableCrateId, + ) -> (DepGraph, Option) { +- if sess.opts.incremental.is_none() { ++ if (*sess.opts.read_incremental()).is_none() { + return (DepGraph::new_disabled(), None); + } + +diff --git a/compiler/rustc_incremental/src/persist/work_product.rs b/compiler/rustc_incremental/src/persist/work_product.rs +index 0aaa9aa8e..4336e32f1 100644 +--- a/compiler/rustc_incremental/src/persist/work_product.rs ++++ b/compiler/rustc_incremental/src/persist/work_product.rs +@@ -24,7 +24,7 @@ pub fn copy_cgu_workproduct_to_incr_comp_cache_dir( + files: &[(&'static str, &Path)], + ) -> (WorkProductId, WorkProduct) { + debug!(?cgu_name, ?files); +- assert!(sess.opts.incremental.is_some()); ++ assert!((*sess.opts.read_incremental()).is_some()); + + let mut saved_files = UnordMap::default(); + for (ext, path) in files { diff --git a/compiler/rustc_infer/src/infer/relate/higher_ranked.rs b/compiler/rustc_infer/src/infer/relate/higher_ranked.rs index 324725a07..baa0f2b04 100644 --- a/compiler/rustc_infer/src/infer/relate/higher_ranked.rs @@ -712,27 +1018,56 @@ index 324725a07..baa0f2b04 100644 } diff --git a/compiler/rustc_interface/src/interface.rs b/compiler/rustc_interface/src/interface.rs -index 7a6a49f37..3df272fa9 100644 +index 7a6a49f37..f31a0f1c7 100644 --- a/compiler/rustc_interface/src/interface.rs +++ b/compiler/rustc_interface/src/interface.rs -@@ -398,7 +398,7 @@ pub fn run_compiler(config: Config, f: impl FnOnce(&Compiler) -> R + Se +@@ -380,11 +380,11 @@ pub fn run_compiler(config: Config, f: impl FnOnce(&Compiler) -> R + Se + trace!("run_compiler"); + + // Set parallel mode before thread pool creation, which will create `Lock`s. +- rustc_data_structures::sync::set_dyn_thread_safe_mode(config.opts.jobs.frontend.is_some()); ++ rustc_data_structures::sync::set_dyn_thread_safe_mode((*config.opts.read_jobs()).frontend.is_some()); + + // Initialize jobserver as early as possible. +- let early_dcx = EarlyDiagCtxt::new(config.opts.error_format); +- let jobs = config.opts.jobs; ++ let early_dcx = EarlyDiagCtxt::new((*config.opts.read_error_format())); ++ let jobs = (*config.opts.read_jobs()); + if let Some(limit) = jobs.frontend.max(jobs.backend).max(jobs.linker.limit()) { + jobserver::initialize(limit.get(), |err| { + let note = "the build environment is likely misconfigured"; +@@ -397,8 +397,8 @@ pub fn run_compiler(config: Config, f: impl FnOnce(&Compiler) -> R + Se + let target = config::build_target_config( &early_dcx, &config.opts.target_triple, - config.opts.sysroot.path(), +- config.opts.sysroot.path(), - config.opts.unstable_opts.unstable_options, ++ (*config.opts.read_sysroot()).path(), + (*config.opts.unstable_opts.read_unstable_options()), ); let file_loader = config.file_loader.unwrap_or_else(|| Box::new(RealFileLoader)); let path_mapping = config.opts.file_path_mapping(); -@@ -416,7 +416,7 @@ pub fn run_compiler(config: Config, f: impl FnOnce(&Compiler) -> R + Se +@@ -414,9 +414,9 @@ pub fn run_compiler(config: Config, f: impl FnOnce(&Compiler) -> R + Se + |current_gcx| { + // The previous `early_dcx` can't be reused here because it doesn't // impl `Send`. Creating a new one is fine. - let early_dcx = EarlyDiagCtxt::new(config.opts.error_format); +- let early_dcx = EarlyDiagCtxt::new(config.opts.error_format); ++ let early_dcx = EarlyDiagCtxt::new((*config.opts.read_error_format())); - let temps_dir = config.opts.unstable_opts.temps_dir.as_deref().map(PathBuf::from); + let temps_dir = (*config.opts.unstable_opts.read_temps_dir()).as_deref().map(PathBuf::from); let early_sess = rustc_session::build_early_session(config.opts, target, config.ice_file); +@@ -424,7 +424,7 @@ pub fn run_compiler(config: Config, f: impl FnOnce(&Compiler) -> R + Se + let mut codegen_backend = match config.make_codegen_backend { + None => util::get_codegen_backend( + &early_dcx, +- &early_sess.opts.sysroot, ++ &(*early_sess.opts.read_sysroot()), + early_sess.opts.unstable_opts.codegen_backend.as_deref(), + &early_sess.target, + ), diff --git a/compiler/rustc_interface/src/passes.rs b/compiler/rustc_interface/src/passes.rs index 8975f01de..9245d5d45 100644 --- a/compiler/rustc_interface/src/passes.rs @@ -814,9 +1149,18 @@ index 8975f01de..9245d5d45 100644 } diff --git a/compiler/rustc_interface/src/queries.rs b/compiler/rustc_interface/src/queries.rs -index 759297bc6..60516087d 100644 +index 759297bc6..4a9bd2d86 100644 --- a/compiler/rustc_interface/src/queries.rs +++ b/compiler/rustc_interface/src/queries.rs +@@ -35,7 +35,7 @@ pub fn codegen_and_build_linker( + Linker { + dep_graph: tcx.dep_graph.clone(), + output_filenames: Arc::clone(tcx.output_filenames(())), +- crate_hash: if tcx.sess.opts.incremental.is_some() { ++ crate_hash: if (*tcx.sess.opts.read_incremental()).is_some() { + Some(tcx.crate_hash(LOCAL_CRATE)) + } else { + None @@ -67,7 +67,7 @@ pub fn link( } }); @@ -826,6 +1170,15 @@ index 759297bc6..60516087d 100644 codegen_backend.print_pass_timings() } +@@ -93,7 +93,7 @@ pub fn link( + + sess.timings.end_section(sess.dcx(), TimingSection::Codegen); + +- if sess.opts.incremental.is_some() ++ if (*sess.opts.read_incremental()).is_some() + && let Some(path) = self.metadata.path() + { + let (id, product) = rustc_incremental::copy_cgu_workproduct_to_incr_comp_cache_dir( diff --git a/compiler/rustc_interface/src/tests.rs b/compiler/rustc_interface/src/tests.rs index 34ac92fae..f0a43672b 100644 --- a/compiler/rustc_interface/src/tests.rs @@ -879,10 +1232,40 @@ index d58b7bfdd..6e8f6570c 100644 sess.opts.output_types.clone(), ) } +diff --git a/compiler/rustc_lint/src/internal.rs b/compiler/rustc_lint/src/internal.rs +index b32fa2515..a794ca64c 100644 +--- a/compiler/rustc_lint/src/internal.rs ++++ b/compiler/rustc_lint/src/internal.rs +@@ -667,7 +667,7 @@ fn is_whitelisted(crate_name: &str) -> bool { + + if let ast::ItemKind::ExternCrate(original_name, imported_name) = &item.kind { + let name = original_name.as_ref().unwrap_or(&imported_name.name).as_str(); +- let externs = &cx.builder.sess().opts.externs; ++ let externs = &(*cx.builder.sess().opts.read_externs()); + if externs.get(name).is_none() && !is_whitelisted(name) { + cx.emit_span_lint( + IMPLICIT_SYSROOT_CRATE_IMPORT, diff --git a/compiler/rustc_metadata/src/creader.rs b/compiler/rustc_metadata/src/creader.rs -index 10693448e..70fa9ba41 100644 +index 10693448e..e28ad214f 100644 --- a/compiler/rustc_metadata/src/creader.rs +++ b/compiler/rustc_metadata/src/creader.rs +@@ -250,14 +250,14 @@ fn set_crate_data(&mut self, cnum: CrateNum, data: CrateMetadata) { + + /// Save the name used to resolve the extern crate in the local crate + /// +- /// The name isn't always the crate's own name, because `sess.opts.externs` can assign it another name. ++ /// The name isn't always the crate's own name, because `(*sess.opts.read_externs())` can assign it another name. + /// It's also not always the same as the `DefId`'s symbol due to renames `extern crate resolved_name as defid_name`. + pub(crate) fn set_resolved_extern_crate_name(&mut self, name: Symbol, extern_crate: CrateNum) { + self.resolved_externs.insert(name, extern_crate); + } + + /// Crate resolved and loaded via the given extern name +- /// (corresponds to names in `sess.opts.externs`) ++ /// (corresponds to names in `(*sess.opts.read_externs())`) + /// + /// May be `None` if the crate wasn't used + pub fn resolved_extern_crate(&self, externs_name: Symbol) -> Option { @@ -299,23 +299,33 @@ pub(crate) fn crate_dependencies_in_postorder(&self, cnum: CrateNum) -> IndexSet deps } @@ -917,6 +1300,15 @@ index 10693448e..70fa9ba41 100644 self.has_alloc_error_handler } +@@ -324,7 +334,7 @@ pub fn had_extern_crate_load_failure(&self) -> bool { + } + + pub fn report_unused_deps(&self, tcx: TyCtxt<'_>) { +- let json_unused_externs = tcx.sess.opts.json_unused_externs; ++ let json_unused_externs = (*tcx.sess.opts.read_json_unused_externs()); + + // We put the check for the option before the lint_level_at_node call + // because the call mutates internal state and introducing it @@ -348,7 +358,7 @@ fn report_target_modifiers_extended( dep_mods: &TargetModifiers, data: &CrateMetadata, @@ -935,6 +1327,51 @@ index 10693448e..70fa9ba41 100644 if !OptionsTargetModifiers::is_target_modifier(flag_name) { tcx.dcx().emit_err(diagnostics::UnknownTargetModifierUnsafeAllowed { flag_name: flag_name.clone(), +@@ -596,7 +606,7 @@ fn register_crate<'tcx>( + let crate_root = metadata.get_root(); + let unhashed = metadata.get_root_unhashed(); + let host_hash = host_lib.as_ref().map(|lib| lib.metadata.get_crate_hash()); +- let private_dep = self.is_private_dep(&tcx.sess.opts.externs, name, private_dep); ++ let private_dep = self.is_private_dep(&(*tcx.sess.opts.read_externs()), name, private_dep); + + // Claim this crate number and cache it + let feed = self.intern_stable_crate_id(tcx, &crate_root)?; +@@ -842,7 +852,7 @@ fn maybe_resolve_crate<'b, 'tcx>( + // not specified by `--extern` on command line parameters, it may be + // `private-dependency` when `register_crate` is called for the first time. Then it must be updated to + // `public-dependency` here. +- let private_dep = self.is_private_dep(&tcx.sess.opts.externs, name, private_dep); ++ let private_dep = self.is_private_dep(&(*tcx.sess.opts.read_externs()), name, private_dep); + let cdata = self.get_crate_data_mut(cnum); + if cdata.is_proc_macro_crate() { + dep_kind = CrateDepKind::MacrosOnly; +@@ -1195,7 +1205,7 @@ fn inject_allocator_crate(&mut self, tcx: TyCtxt<'_>, krate: &ast::Crate) { + } + + fn inject_forced_externs(&mut self, tcx: TyCtxt<'_>) { +- for (name, entry) in tcx.sess.opts.externs.iter() { ++ for (name, entry) in (*tcx.sess.opts.read_externs()).iter() { + if entry.force { + let name_interned = Symbol::intern(name); + if !self.used_extern_options.contains(&name_interned) { +@@ -1253,7 +1263,7 @@ fn report_unused_deps_in_crate(&mut self, tcx: TyCtxt<'_>, krate: &ast::Crate) { + // Make a point span rather than covering the whole file + let span = krate.spans.inner_span.shrink_to_lo(); + // Complain about anything left over +- for (name, entry) in tcx.sess.opts.externs.iter() { ++ for (name, entry) in (*tcx.sess.opts.read_externs()).iter() { + if let ExternLocation::FoundInLibrarySearchDirectories = entry.location { + // Don't worry about pathless `--extern foo` sysroot references + continue; +@@ -1268,7 +1278,7 @@ fn report_unused_deps_in_crate(&mut self, tcx: TyCtxt<'_>, krate: &ast::Crate) { + } + + // Got a real unused --extern +- if tcx.sess.opts.json_unused_externs.is_enabled() { ++ if (*tcx.sess.opts.read_json_unused_externs()).is_enabled() { + self.unused_externs.push(name_interned); + continue; + } diff --git a/compiler/rustc_metadata/src/dependency_format.rs b/compiler/rustc_metadata/src/dependency_format.rs index 4a6c23a10..7ab211e3a 100644 --- a/compiler/rustc_metadata/src/dependency_format.rs @@ -1008,7 +1445,7 @@ index 6cb6ad001..8586b036d 100644 diagnostics::StaticLinkingNotSupported::UserRequested { lib_name: lib.name, diff --git a/compiler/rustc_metadata/src/rmeta/encoder.rs b/compiler/rustc_metadata/src/rmeta/encoder.rs -index 989ca8361..0bba24a5c 100644 +index 39bbdc31f..c708227a1 100644 --- a/compiler/rustc_metadata/src/rmeta/encoder.rs +++ b/compiler/rustc_metadata/src/rmeta/encoder.rs @@ -589,6 +589,8 @@ fn encode_source_map( @@ -1048,7 +1485,7 @@ index 989ca8361..0bba24a5c 100644 // Encode all the entries and extra information in the crate, // culminating in the `CrateRoot` which points to all of it. diff --git a/compiler/rustc_middle/src/dep_graph/graph.rs b/compiler/rustc_middle/src/dep_graph/graph.rs -index dcb5775f2..211d1d6f5 100644 +index dcb5775f2..58f25bbc9 100644 --- a/compiler/rustc_middle/src/dep_graph/graph.rs +++ b/compiler/rustc_middle/src/dep_graph/graph.rs @@ -390,7 +390,11 @@ pub fn with_task<'tcx, OP, R>( @@ -1071,6 +1508,15 @@ index dcb5775f2..211d1d6f5 100644 )); let result = with_deps(TaskDepsRef::Allow(&task_deps), op); let task_deps = task_deps.into_inner(); +@@ -676,7 +681,7 @@ fn assert_dep_node_not_yet_allocated_in_current_session( + let ok = match color { + DepNodeColor::Unknown => true, + DepNodeColor::Red => false, +- DepNodeColor::Green(..) => sess.opts.jobs.frontend.is_some(), // Other threads may mark this green ++ DepNodeColor::Green(..) => (*sess.opts.read_jobs()).frontend.is_some(), // Other threads may mark this green + }; + if !ok { + panic!("{}", msg()) @@ -1311,15 +1316,25 @@ pub struct TaskDeps { node: Option, @@ -1223,6 +1669,19 @@ index 1c476fc91..4a76764f4 100644 GraphEncoder { status, retained_graph, profiler: sess.prof.clone() } } +diff --git a/compiler/rustc_middle/src/hir/map.rs b/compiler/rustc_middle/src/hir/map.rs +index ea266e003..e889076b0 100644 +--- a/compiler/rustc_middle/src/hir/map.rs ++++ b/compiler/rustc_middle/src/hir/map.rs +@@ -1249,7 +1249,7 @@ fn legacy_crate_hash(tcx: TyCtxt<'_>) -> Svh { + upstream_crates.stable_hash(&mut hcx, &mut stable_hasher); + source_file_names.stable_hash(&mut hcx, &mut stable_hasher); + debugger_visualizers.stable_hash(&mut hcx, &mut stable_hasher); +- if tcx.sess.opts.incremental.is_some() { ++ if (*tcx.sess.opts.read_incremental()).is_some() { + let definitions = tcx.untracked().definitions.freeze(); + let mut owner_spans: Vec<_> = tcx + .hir_crate_items(()) diff --git a/compiler/rustc_middle/src/lint.rs b/compiler/rustc_middle/src/lint.rs index 8822ffee6..2cccc2a36 100644 --- a/compiler/rustc_middle/src/lint.rs @@ -1362,9 +1821,18 @@ index ae30093b1..5ceeb0c88 100644 } // Write allocation bytes. diff --git a/compiler/rustc_middle/src/mono.rs b/compiler/rustc_middle/src/mono.rs -index cf567fa1c..82b2c8ef3 100644 +index cf567fa1c..a912cab0c 100644 --- a/compiler/rustc_middle/src/mono.rs +++ b/compiler/rustc_middle/src/mono.rs +@@ -177,7 +177,7 @@ pub fn instantiation_mode(&self, tcx: TyCtxt<'tcx>) -> InstantiationMode { + if tcx.sess.opts.optimize == OptLevel::No { + return InstantiationMode::GloballyShared { may_conflict: false }; + } +- if tcx.sess.opts.incremental.is_none() { ++ if (*tcx.sess.opts.read_incremental()).is_none() { + return InstantiationMode::LocalCopy; + } + return opt_incr_drop_glue_mode(tcx, ty); @@ -568,7 +568,7 @@ fn item_sort_key<'tcx>(tcx: TyCtxt<'tcx>, item: MonoItem<'tcx>) -> ItemSortKey<' } @@ -1375,11 +1843,15 @@ index cf567fa1c..82b2c8ef3 100644 // is already deterministic. // diff --git a/compiler/rustc_middle/src/ty/context.rs b/compiler/rustc_middle/src/ty/context.rs -index aa7fe0872..e56a7d587 100644 +index aa7fe0872..dd39a1105 100644 --- a/compiler/rustc_middle/src/ty/context.rs +++ b/compiler/rustc_middle/src/ty/context.rs -@@ -1155,7 +1155,7 @@ pub fn needs_hir_hash(self) -> bool { - || self.sess.opts.incremental.is_some() +@@ -1152,10 +1152,10 @@ pub fn needs_hir_hash(self) -> bool { + // for similar source builds (may change in the future, this is part + // of the proof of concept impl for the metrics initiative project goal) + cfg!(debug_assertions) +- || self.sess.opts.incremental.is_some() ++ || (*self.sess.opts.read_incremental()).is_some() || self.needs_metadata() || self.sess.instrument_coverage() - || self.sess.opts.unstable_opts.metrics_dir.is_some() @@ -1387,6 +1859,15 @@ index aa7fe0872..e56a7d587 100644 } /// Whether the combined per-owner HIR hash (`OwnerInfo::opt_hash`, which folds `parenting`, +@@ -1175,7 +1175,7 @@ pub fn needs_hir_hash(self) -> bool { + pub fn needs_owner_info_hash(self) -> bool { + self.needs_hir_hash() + && (!self.sess.opts.unstable_opts.metadata_crate_hash +- || self.sess.opts.incremental.is_some() ++ || (*self.sess.opts.read_incremental()).is_some() + || cfg!(debug_assertions)) + } + @@ -2805,7 +2805,7 @@ pub fn is_const_trait_impl(self, def_id: DefId) -> bool { } @@ -1409,6 +1890,19 @@ index fb4e30b16..c6a883286 100644 return regular; } +diff --git a/compiler/rustc_middle/src/ty/mod.rs b/compiler/rustc_middle/src/ty/mod.rs +index 1f2ffb17c..8f20853b7 100644 +--- a/compiler/rustc_middle/src/ty/mod.rs ++++ b/compiler/rustc_middle/src/ty/mod.rs +@@ -2336,7 +2336,7 @@ pub fn fn_abi_of_instance( + ) -> Result<&'tcx FnAbi<'tcx, Ty<'tcx>>, &'tcx FnAbiError<'tcx>> { + // Only deduce attrs in full, optimized builds. Otherwise, avoid the query system overhead + // of ever invoking the `fn_abi_of_instance_raw` query. +- if self.sess.opts.optimize != OptLevel::No && self.sess.opts.incremental.is_none() { ++ if self.sess.opts.optimize != OptLevel::No && (*self.sess.opts.read_incremental()).is_none() { + self.fn_abi_of_instance_raw(query) + } else { + self.fn_abi_of_instance_no_deduced_attrs(query) diff --git a/compiler/rustc_middle/src/ty/print/pretty.rs b/compiler/rustc_middle/src/ty/print/pretty.rs index cafe93c59..c2bc8681b 100644 --- a/compiler/rustc_middle/src/ty/print/pretty.rs @@ -1558,6 +2052,45 @@ index 1e63dca78..e42069318 100644 let mut vis = EnsureCoroutineFieldAssignmentsNeverAlias { assigned_local: None, saved_locals: &liveness_info.saved_locals, +diff --git a/compiler/rustc_mir_transform/src/cross_crate_inline.rs b/compiler/rustc_mir_transform/src/cross_crate_inline.rs +index 53d58439b..01618985e 100644 +--- a/compiler/rustc_mir_transform/src/cross_crate_inline.rs ++++ b/compiler/rustc_mir_transform/src/cross_crate_inline.rs +@@ -82,7 +82,7 @@ fn cross_crate_inlinable(tcx: TyCtxt<'_>, def_id: LocalDefId) -> bool { + + // Don't do any inference when incremental compilation is enabled; the additional inlining that + // inference permits also creates more work for small edits. +- if tcx.sess.opts.incremental.is_some() { ++ if (*tcx.sess.opts.read_incremental()).is_some() { + return false; + } + +diff --git a/compiler/rustc_mir_transform/src/deduce_param_attrs.rs b/compiler/rustc_mir_transform/src/deduce_param_attrs.rs +index 8814670ca..33c95b713 100644 +--- a/compiler/rustc_mir_transform/src/deduce_param_attrs.rs ++++ b/compiler/rustc_mir_transform/src/deduce_param_attrs.rs +@@ -177,7 +177,7 @@ pub(super) fn deduced_param_attrs<'tcx>( + ) -> &'tcx [DeducedParamAttrs] { + // This computation is unfortunately rather expensive, so don't do it unless we're optimizing. + // Also skip it in incremental mode. +- if tcx.sess.opts.optimize == OptLevel::No || tcx.sess.opts.incremental.is_some() { ++ if tcx.sess.opts.optimize == OptLevel::No || (*tcx.sess.opts.read_incremental()).is_some() { + return &[]; + } + +diff --git a/compiler/rustc_mir_transform/src/inline.rs b/compiler/rustc_mir_transform/src/inline.rs +index 38021f4b7..ad74f65bd 100644 +--- a/compiler/rustc_mir_transform/src/inline.rs ++++ b/compiler/rustc_mir_transform/src/inline.rs +@@ -55,7 +55,7 @@ fn policy(&self, ctx: &crate::PassCtx<'_>) -> PassPolicy { + 2 => { + (ctx.opts.optimize == OptLevel::More + || ctx.opts.optimize == OptLevel::Aggressive) +- && ctx.opts.incremental.is_none() ++ && (*ctx.opts.read_incremental()).is_none() + } + _ => true, + }), diff --git a/compiler/rustc_mir_transform/src/pass_manager.rs b/compiler/rustc_mir_transform/src/pass_manager.rs index f3f65a070..8ecc26910 100644 --- a/compiler/rustc_mir_transform/src/pass_manager.rs @@ -1621,9 +2154,45 @@ index bfe3ba91b..7b204f959 100644 && tcx.is_closure_like(def_id) { diff --git a/compiler/rustc_monomorphize/src/partitioning.rs b/compiler/rustc_monomorphize/src/partitioning.rs -index 16df895b0..75376ac08 100644 +index 16df895b0..7fc1f98a8 100644 --- a/compiler/rustc_monomorphize/src/partitioning.rs +++ b/compiler/rustc_monomorphize/src/partitioning.rs +@@ -203,7 +203,7 @@ fn place_mono_items<'tcx, I>(cx: &PartitioningCx<'_, 'tcx>, mono_items: I) -> Pl + I: Iterator>, + { + let mut codegen_units = UnordMap::default(); +- let is_incremental_build = cx.tcx.sess.opts.incremental.is_some(); ++ let is_incremental_build = (*cx.tcx.sess.opts.read_incremental()).is_some(); + let mut internalization_candidates = UnordSet::default(); + + // Determine if monomorphizations instantiated in this crate will be made +@@ -397,7 +397,7 @@ fn merge_codegen_units<'tcx>( + // the `compiler_builtins` crate sets `codegen-units = 10000` and it's + // critical they aren't merged. Also, some tests use explicit small values + // and likewise won't work if small CGUs are merged. +- while cx.tcx.sess.opts.incremental.is_none() ++ while (*cx.tcx.sess.opts.read_incremental()).is_none() + && matches!(cx.tcx.sess.codegen_units(), CodegenUnits::Default(_)) + && codegen_units.len() > 1 + && codegen_units.iter().any(|cgu| cgu.size_estimate() < NON_INCR_MIN_CGU_SIZE) +@@ -420,7 +420,7 @@ fn merge_codegen_units<'tcx>( + let cgu_name_builder = &mut CodegenUnitNameBuilder::new(cx.tcx); + + // Rename the newly merged CGUs. +- if cx.tcx.sess.opts.incremental.is_some() { ++ if (*cx.tcx.sess.opts.read_incremental()).is_some() { + // If we are doing incremental compilation, we want CGU names to + // reflect the path of the source level module they correspond to. + // For CGUs that contain the code of multiple modules because of the +@@ -675,7 +675,7 @@ fn characteristic_def_id_of_mono_item<'tcx>( + + if let Some((impl_def_id, DefKind::Impl { of_trait })) = assoc_parent { + if of_trait +- && tcx.sess.opts.incremental.is_some() ++ && (*tcx.sess.opts.read_incremental()).is_some() + && tcx.is_lang_item(tcx.impl_trait_id(impl_def_id), LangItem::Drop) + { + // Put `Drop::drop` into the same cgu as `drop_glue` @@ -1221,7 +1221,7 @@ fn collect_and_partition_mono_items(tcx: TyCtxt<'_>, (): ()) -> MonoItemPartitio tcx.dcx().emit_fatal(CouldntDumpMonoStats { error: err.to_string() }); } @@ -1643,9 +2212,18 @@ index 16df895b0..75376ac08 100644 let filename = format!("{crate_name}.mono_items.{ext}"); let output_path = output_directory.join(&filename); diff --git a/compiler/rustc_query_impl/src/execution.rs b/compiler/rustc_query_impl/src/execution.rs -index ca60614ba..e3bf3285f 100644 +index ca60614ba..91a4e5778 100644 --- a/compiler/rustc_query_impl/src/execution.rs +++ b/compiler/rustc_query_impl/src/execution.rs +@@ -292,7 +292,7 @@ fn try_execute_query<'tcx, C: QueryCache, const INCR: bool>( + // re-executing the query since `try_start` only checks that the query is not currently + // executing, but another thread may have already completed the query and stores it result + // in the query cache. +- if tcx.sess.opts.jobs.frontend.is_some() { ++ if (*tcx.sess.opts.read_jobs()).frontend.is_some() { + if let Some((value, index)) = query.cache.lookup(&key) { + tcx.prof.query_cache_hit(index.into()); + return (value, Some(index)); @@ -543,7 +543,7 @@ fn load_from_disk_or_invoke_provider_green<'tcx, C: QueryCache>( }; let (value, verify) = match try_value { @@ -1677,6 +2255,28 @@ index afcab4a0b..f2a75b6a1 100644 } /// Inner implementation of [`DepKindVTable::promote_from_disk_fn`] for queries. +diff --git a/compiler/rustc_resolve/src/diagnostics/impls.rs b/compiler/rustc_resolve/src/diagnostics/impls.rs +index 744e7c222..86c267841 100644 +--- a/compiler/rustc_resolve/src/diagnostics/impls.rs ++++ b/compiler/rustc_resolve/src/diagnostics/impls.rs +@@ -2370,7 +2370,7 @@ fn decl_description(&self, b: Decl<'_>, ident: Ident, scope: Scope<'_>) -> Strin + let (built_in, from) = match scope { + Scope::StdLibPrelude | Scope::MacroUsePrelude => ("", " from prelude"), + Scope::ExternPreludeFlags +- if self.tcx.sess.opts.externs.get(ident.as_str()).is_some() ++ if (*self.tcx.sess.opts.read_externs()).get(ident.as_str()).is_some() + || matches!(res, Res::OpenMod(..)) => + { + ("", " passed with `--extern`") +@@ -3149,7 +3149,7 @@ pub(crate) fn report_path_resolution_error( + None, + ) + } else if self.tcx.sess.is_rust_2015() { +- let crate_is_available = self.tcx.sess.opts.externs.get(ident.as_str()).is_some(); ++ let crate_is_available = (*self.tcx.sess.opts.read_externs()).get(ident.as_str()).is_some(); + let (suggestion_message, help) = if crate_is_available { + let edition_help = format!( + "if you're trying to use a dependency named `{ident}`, upgrade your \ diff --git a/compiler/rustc_session/src/config.rs b/compiler/rustc_session/src/config.rs index 64cfdfd5f..7c3993f87 100644 --- a/compiler/rustc_session/src/config.rs @@ -1738,10 +2338,37 @@ index e3580d36e..b807713e9 100644 #![feature(rustc_attrs)] // To generate CodegenOptionsTargetModifiers and UnstableOptionsTargetModifiers enums diff --git a/compiler/rustc_session/src/options.rs b/compiler/rustc_session/src/options.rs -index 9a3e74c28..264c4c123 100644 +index 9a3e74c28..293aaee62 100644 --- a/compiler/rustc_session/src/options.rs +++ b/compiler/rustc_session/src/options.rs -@@ -543,6 +543,26 @@ pub struct $struct_name { +@@ -281,6 +281,26 @@ pub struct Options { + pub mitigation_coverage_map: mitigation_coverage::MitigationCoverageMap, + } + ++ impl Options { ++ $( ++ /// The option's value. A read of an option the dependency graph does not ++ /// track is reported, for testing incremental compilation ++ /// (`rustc_data_structures::untracked`). ++ #[inline] ++ #[track_caller] ++ #[allow(dead_code)] ++ pub fn ${concat(read_, $opt)}(&self) -> &$t { ++ if stringify!($dep_tracking_marker) == "UNTRACKED" { ++ rustc_data_structures::untracked::untracked_read(concat!( ++ "the untracked option ", ++ stringify!($opt) ++ )); ++ } ++ &self.$opt ++ } ++ )* ++ } ++ + impl Options { + pub fn dep_tracking_hash(&self, for_crate_hash: bool) -> Hash64 { + let mut sub_hashes = BTreeMap::new(); +@@ -543,6 +563,26 @@ pub struct $struct_name { )* } @@ -1782,7 +2409,7 @@ index 4824abae6..ba42fc461 100644 match crate_type { CrateType::Rlib => { diff --git a/compiler/rustc_session/src/session.rs b/compiler/rustc_session/src/session.rs -index f068aaa58..447a2d9a6 100644 +index f068aaa58..c15294494 100644 --- a/compiler/rustc_session/src/session.rs +++ b/compiler/rustc_session/src/session.rs @@ -352,11 +352,11 @@ pub fn early_lto(&self) -> LtoCli { @@ -1814,16 +2441,39 @@ index f068aaa58..447a2d9a6 100644 || self.prof.is_args_recording_enabled() || self.opts.output_types.contains_key(&OutputType::Mir) || std::env::var_os("RUSTC_LOG").is_some() -@@ -897,7 +897,7 @@ pub fn diagnostic_width(&self) -> usize { +@@ -687,7 +687,10 @@ pub fn record_trimmed_def_paths(&self) { + self.dcx().set_must_produce_diag() + } + ++ #[track_caller] + pub fn proc_macro_quoted_spans(&self) -> impl Iterator { ++ // Recorded while expanding macros, before any query; not tracked. ++ rustc_data_structures::untracked::untracked_read("proc macro quoted spans"); + // This is equivalent to `.iter().copied().enumerate()`, but that isn't possible for + // AppendOnlyVec, so we resort to this scheme. + self.proc_macro_quoted_spans.iter_enumerated() +@@ -895,9 +898,9 @@ pub fn emit_lifetime_markers(&self) -> bool { + + pub fn diagnostic_width(&self) -> usize { let default_column_width = 140; - if let Some(width) = self.opts.diagnostic_width { +- if let Some(width) = self.opts.diagnostic_width { ++ if let Some(width) = (*self.opts.read_diagnostic_width()) { width - } else if self.opts.unstable_opts.ui_testing { + } else if (*self.opts.unstable_opts.read_ui_testing()) { default_column_width } else { termize::dimensions().map_or(default_column_width, |(w, _)| w) -@@ -1070,7 +1070,7 @@ pub fn fewer_names(&self) -> bool { +@@ -1029,7 +1032,7 @@ pub fn lto(&self) -> config::Lto { + + // If processing command line options determined that we're incompatible + // with ThinLTO (e.g., `-C lto --emit llvm-ir`) then return that option. +- if self.opts.cli_forced_local_thinlto_off { ++ if (*self.opts.read_cli_forced_local_thinlto_off()) { + return config::Lto::No; + } + +@@ -1070,7 +1073,7 @@ pub fn fewer_names(&self) -> bool { } pub fn unstable_options(&self) -> bool { @@ -1832,7 +2482,25 @@ index f068aaa58..447a2d9a6 100644 } pub fn is_nightly_build(&self) -> bool { -@@ -1280,9 +1280,9 @@ pub fn pointer_authentication_init_fini(&self) -> Option<&PointerAuthSchema> { +@@ -1181,7 +1184,7 @@ pub fn must_emit_unwind_tables(&self) -> bool { + /// Returns the number of codegen units that should be used for this + /// compilation + pub fn codegen_units(&self) -> CodegenUnits { +- if let Some(n) = self.opts.cli_forced_codegen_units { ++ if let Some(n) = (*self.opts.read_cli_forced_codegen_units()) { + return CodegenUnits::User(n); + } + if let Some(n) = self.target.default_codegen_units { +@@ -1191,7 +1194,7 @@ pub fn codegen_units(&self) -> CodegenUnits { + // If incremental compilation is turned on, we default to a high number + // codegen units in order to reduce the "collateral damage" small + // changes cause. +- if self.opts.incremental.is_some() { ++ if (*self.opts.read_incremental()).is_some() { + return CodegenUnits::Default(256); + } + +@@ -1280,9 +1283,9 @@ pub fn pointer_authentication_init_fini(&self) -> Option<&PointerAuthSchema> { // JUSTIFICATION: part of session construction #[allow(rustc::bad_opt_access)] fn default_emitter(sopts: &config::Options, source_map: Arc) -> Box { @@ -1845,7 +2513,7 @@ index f068aaa58..447a2d9a6 100644 TerminalUrl::Auto => { match (std::env::var("COLORTERM").as_deref(), std::env::var("TERM").as_deref()) { (Ok("truecolor"), Ok("xterm-256color")) -@@ -1310,9 +1310,9 @@ fn default_emitter(sopts: &config::Options, source_map: Arc) -> Box) -> Box Box::new( -@@ -1323,9 +1323,9 @@ fn default_emitter(sopts: &config::Options, source_map: Arc) -> Box) -> Box Some(Arc::new(profiler)), -@@ -1438,7 +1438,7 @@ pub fn build_session( +@@ -1438,7 +1441,7 @@ pub fn build_session( let prof = SelfProfilerRef::new( self_profiler, @@ -1907,7 +2575,7 @@ index f068aaa58..447a2d9a6 100644 ); let ctfe_backtrace = Lock::new(match env::var("RUSTC_CTFE_BACKTRACE") { -@@ -1759,7 +1759,7 @@ fn validate_commandline_args_with_session_available(sess: &Session) { +@@ -1759,7 +1762,7 @@ fn validate_commandline_args_with_session_available(sess: &Session) { } if !sess.target.options.supported_split_debuginfo.contains(&sess.split_debuginfo()) @@ -1916,7 +2584,7 @@ index f068aaa58..447a2d9a6 100644 { sess.dcx().emit_err(diagnostics::SplitDebugInfoUnstablePlatform { debuginfo: sess.split_debuginfo(), -@@ -1794,7 +1794,7 @@ fn validate_commandline_args_with_session_available(sess: &Session) { +@@ -1794,7 +1797,7 @@ fn validate_commandline_args_with_session_available(sess: &Session) { sess.dcx().emit_err(diagnostics::InstrumentationNotSupported { us: "XRay".to_string() }); } diff --git a/docs/hunt/repro.sh b/docs/hunt/repro.sh index dbd520c..a2948b8 100755 --- a/docs/hunt/repro.sh +++ b/docs/hunt/repro.sh @@ -6,6 +6,7 @@ here=$(cd "$(dirname "$0")" && pwd) rustc=${RUSTC:-$(rustc +nightly-2026-10-06 --print sysroot)/bin/rustc} d=$(mktemp -d) trap 'rm -rf "$d"' EXIT +cd "$d" # rustc writes ICE reports to the working directory rc() { "$rustc" --edition 2024 "$@" 2> "$d/err" || { cat "$d/err"; exit 1; }; } echo -n "threads-rpitit, 8 builds with -Zthreads=8, distinct .rmeta files: " @@ -90,3 +91,54 @@ echo 'pub fn g(v: &[T]) -> Option<&[T]> { v.get(..3) }' >> "$d/ia/lib.rs" "$rustc" $iaf -Cincremental="$d/i13" --out-dir "$d/ia/o1" "$d/ia/lib.rs" 2> /dev/null "$rustc" $iaf -Cincremental="$d/i14" --out-dir "$d/ia/o2" "$d/ia/lib.rs" 2> /dev/null cmp -s "$d/ia/o1/libx.rmeta" "$d/ia/o2/libx.rmeta" && echo same || echo DIFFER + +echo -n "thinlto-order, -Clto=thin -g with incremental, 30 clean builds of a binary, distinct object sets: " +mkdir -p "$d/to" +cat > "$d/to/up.rs" <<'RS' +pub struct Matrix(pub [[u32; C]; R]); +impl Matrix { + pub fn transpose(&self) -> Matrix { + let mut out = [[0; R]; C]; + for i in 0..R { for j in 0..C { out[j][i] = self.0[i][j]; } } + Matrix(out) + } +} +#[inline(always)] +pub fn always_inline(v: u32) -> u32 { v.rotate_left(3) } +RS +cat > "$d/to/bin.rs" <<'RS' +use up::Matrix; +pub fn spin(m: &Matrix<2, 3>) -> Matrix<3, 2> { m.transpose() } +fn main() { let m = Matrix([[1,2,3],[4,5,6]]); println!("{}", spin(&m).0[2][1] + up::always_inline(std::env::args().count() as u32)); } +RS +"$rustc" --edition 2021 --crate-type rlib "$d/to/up.rs" -o "$d/to/libup.rlib" +for i in $(seq 1 30); do + rm -rf "$d/to/o" "$d/to/inc"; mkdir "$d/to/o" + "$rustc" --edition 2021 -g -Clto=thin -Cincremental="$d/to/inc" -Csave-temps --extern up="$d/to/libup.rlib" \ + -o "$d/to/o/bin" "$d/to/bin.rs" 2> /dev/null + cat $(ls "$d"/to/o/*.rcgu.o | grep -v 'no-opt\|thin-lto' | sort) | sha256sum +done | sort -u | wc -l + +echo -n "no-prepopulate-link, a dylib with -Cno-prepopulate-passes -Zshare-generics=no -Zthinlto=yes: " +mkdir -p "$d/np" +echo 'pub fn f(a: &mut u8, b: &mut u8) { core::mem::swap(a, b) }' > "$d/np/a.rs" +if "$rustc" --edition 2021 --crate-type dylib -Ccodegen-units=16 -Cno-prepopulate-passes -Zshare-generics=no \ + -Zthinlto=yes --out-dir "$d/np" "$d/np/a.rs" > "$d/np/log" 2>&1; then echo links +else echo "fails: $(grep -o 'undefined hidden symbol: [^ ]*' "$d/np/log" | head -1)"; fi + +echo -n "print-type-sizes-ice, a session with -Zprint-type-sizes, an edit, a session without: " +mkdir -p "$d/pt" +cat > "$d/pt/lib.rs" <<'RS' +pub async fn inner() -> u32 { 1 } +pub async fn outer() -> u32 { let s = String::from("x"); inner().await + s.len() as u32 } +pub fn make() -> impl std::future::Future { outer() } +RS +"$rustc" --edition 2021 --crate-type lib -Cincremental="$d/pt/inc" -Zprint-type-sizes --out-dir "$d/pt" "$d/pt/lib.rs" > /dev/null 2>&1 +echo 'pub fn g() {}' >> "$d/pt/lib.rs" +if "$rustc" --edition 2021 --crate-type lib -Cincremental="$d/pt/inc" --out-dir "$d/pt" "$d/pt/lib.rs" > "$d/pt/log" 2>&1 +then echo builds; else echo "ICE: $(grep -o 'trimmed_def_paths. called[^.]*' "$d/pt/log")"; fi + +echo -n "rwpi-segfault, -g -Crelocation-model=rwpi on a static mut: " +echo 'pub static mut M: u32 = 0;' > "$d/rw.rs" +"$rustc" --crate-type lib -g -Crelocation-model=rwpi "$d/rw.rs" -o "$d/rw.rlib" > /dev/null 2>&1 +rc=$?; if [ $rc = 0 ]; then echo builds; else echo "exit $rc"; fi diff --git a/docs/hunt/rwpi-debuginfo-segfault.md b/docs/hunt/rwpi-debuginfo-segfault.md new file mode 100644 index 0000000..50c67c9 --- /dev/null +++ b/docs/hunt/rwpi-debuginfo-segfault.md @@ -0,0 +1,36 @@ +# Finding 12: rustc segfaults on `-g -Crelocation-model=rwpi` for x86_64 with a mutable static + +Facts for a report; the report itself is for a person to write (rust-lang/rust's LLM policy). + +**Repro.** `docs/hunt/repro.sh`, "rwpi-segfault": + + echo 'pub static mut M: u32 = 0;' > b.rs + rustc --crate-type lib -g -Crelocation-model=rwpi b.rs + +on x86_64-unknown-linux-gnu. Stable flags only. + +**Expected.** An rlib, or an error that `rwpi` is not supported for this target +(`ropi`/`rwpi` are ARM relocation models; `rustc --print relocation-models` lists them for +x86_64 too). + +**Actual.** `error: rustc interrupted by SIGSEGV, printing backtrace`, exit 139. The +backtrace is in LLVM: `MCStreamer::visitUsedExpr` ← `MCStreamer::emitValue` ← +`DIEValue::emitValue` ← `AsmPrinter::emitDwarfDIE` ← `DwarfDebug::endModule`, on a codegen +worker thread. + +**Conditions.** `-Crelocation-model=rwpi` and `ropi-rwpi` crash; `ropi` does not. Debuginfo +is needed (`-g`). The static must be writable: `static mut`, or a static with interior +mutability (`AtomicU32`); an immutable `static` and a `thread_local!` do not crash. + +**Versions.** Fine on 1.20.0 through 1.59.0 (LLVM 13); crashes on 1.60.0 (LLVM 14) through +1.98.1 and nightly-2026-10-06 (LLVM 23). + +**Cause, as far as followed.** The crash is in emitting the DWARF location of the global: with +RWPI, LLVM describes a writable global's address relative to the static base, and on x86_64 +the expression it emits is null. rustc passes `rwpi` to LLVM for any target. An LLVM commit +about wrong debuginfo for RWPI globals on ARM +(https://repo.hca.bsc.es/gitlab/rferrer/llvm-epi/-/commit/04dc68710ad2b30a1d3b4a2ca33005af2c9460eb) +is in the same area; whether it is the change in LLVM 14 was not checked. + +**How mirth found it.** Trying the hand-picked values of `-Crelocation-model` alone on +`fixtures/sink` (`docs/flags.md`), whose `sink-core` has an `AtomicU32` static. diff --git a/docs/hunt/rwpi-stopgap.patch b/docs/hunt/rwpi-stopgap.patch new file mode 100644 index 0000000..9f97979 --- /dev/null +++ b/docs/hunt/rwpi-stopgap.patch @@ -0,0 +1,16 @@ +--- a/compiler/rustc_session/src/session.rs ++++ b/compiler/rustc_session/src/session.rs +@@ -1748,6 +1754,13 @@ + } + } + ++ // LLVM crashes emitting debuginfo for writable statics with these models on other targets. ++ if matches!(sess.opts.cg.relocation_model, Some(RelocModel::Rwpi | RelocModel::RopiRwpi)) ++ && sess.target.arch != Arch::Arm ++ { ++ sess.dcx().err("`-Crelocation-model=rwpi` and `ropi-rwpi` are only supported on ARM"); ++ } ++ + if sess.opts.unstable_opts.branch_protection.is_some() && sess.target.arch != Arch::AArch64 { + sess.dcx().emit_err(diagnostics::BranchProtectionRequiresAArch64); + } diff --git a/docs/hunt/thinlto-module-order.md b/docs/hunt/thinlto-module-order.md new file mode 100644 index 0000000..d967d63 --- /dev/null +++ b/docs/hunt/thinlto-module-order.md @@ -0,0 +1,43 @@ +# Finding 10: ThinLTO input order follows codegen completion + +Facts for a report; the report itself is for a person to write (rust-lang/rust's LLM policy). + +**Repro.** `docs/hunt/repro.sh`, "thinlto-order": a two-crate binary (an upstream rlib with a +const-generic method and an `#[inline(always)]` function) built 30 times from clean with +`-g -Clto=thin -Cincremental=`. That is Cargo's dev profile with `lto = "thin"`. + +**Expected.** The same object files and binary every time. + +**Actual.** Two distinct sets of object files, roughly one build in five. With rust-lld (the +default linker on x86_64-unknown-linux-gnu since 1.90) the binary differs too: 8 of 30 +builds, nightly-2026-10-06. Linked with BFD (`-Zunstable-options -Clinker-features=-lld`), +the binary is the same. + +**What differs.** `__rustc_debug_gdb_scripts_section__`, which every codegen unit with +debuginfo defines as `linkonce_odr`, ends up kept in a different codegen unit. + +**Conditions.** + +| | objects | binary | +|---|---|---| +| `-g -Clto=thin -Cincremental` | differ | differ (lld) | +| same, `--jobs-backend=1 -Zunstable-options` | same (30 of 30) | same | +| same without `-g` | same | same | +| same without `-Cincremental` | same | same | +| `-g -Clto=thin -Cincremental -Copt-level=1` | same | same | +| `-g -Clto=fat -Cincremental` | same | same | +| `-g -Zthinlto=yes -Cincremental` (local ThinLTO at opt-level 0) | differ | differ | + +**Versions.** Objects differ on 1.60.0, 1.71.0, 1.78.0, 1.87.0, 1.89.0 and the nightly. The +binary differs from 1.90.0 on (rust-lld). + +**Cause.** `rustc_codegen_ssa/src/back/write.rs` pushes each module onto `needs_thin_lto` +as its codegen finishes (`ThinLtoInput::Red`) and never sorts the list. In +`rustc_llvm/llvm-wrapper/PassWrapper.cpp`, the prevailing copy of a linkonce symbol is +`getFirstDefinitionForLinker` of the combined index, which follows that order, so the +module that keeps the definition depends on thread timing. One backend job removes the +difference. + +**How mirth found it.** The transitions walk (`docs/flags.md`) reported a rebuild differing +from a clean build in two rows; repeating clean builds showed clean builds differ too, and a +delta minimization of the flags left `-Zshare-generics=no -Zthinlto=yes`. diff --git a/docs/hunt/thinlto-order-stopgap.patch b/docs/hunt/thinlto-order-stopgap.patch new file mode 100644 index 0000000..6d7cff8 --- /dev/null +++ b/docs/hunt/thinlto-order-stopgap.patch @@ -0,0 +1,29 @@ +--- a/compiler/rustc_codegen_ssa/src/back/write.rs ++++ b/compiler/rustc_codegen_ssa/src/back/write.rs +@@ -782,6 +782,15 @@ + Green { wp: WorkProduct, bitcode_path: PathBuf }, + } + ++impl ThinLtoInput { ++ fn name(&self) -> &str { ++ match self { ++ ThinLtoInput::Red { name, .. } => name, ++ ThinLtoInput::Green { wp, .. } => &wp.cgu_name, ++ } ++ } ++} ++ + /// Actual LTO type we end up choosing based on multiple factors. + pub(crate) enum ComputedLtoType { + No, +@@ -1745,6 +1754,10 @@ + for (bitcode_path, wp) in lto_import_only_modules { + needs_thin_lto.push(ThinLtoInput::Green { wp, bitcode_path }) + } ++ // Modules arrive in the order their codegen finished. ThinLTO keeps the first ++ // definition of a linkonce symbol, so sort to make the output independent of ++ // thread timing. ++ needs_thin_lto.sort_by(|a, b| a.name().cmp(b.name())); + + if cgcx.lto == Lto::ThinLocal { + compiled_modules.extend(do_thin_lto::( diff --git a/docs/hunt/verify-reuse.patch b/docs/hunt/verify-reuse.patch index 1a1483d..6e96153 100644 --- a/docs/hunt/verify-reuse.patch +++ b/docs/hunt/verify-reuse.patch @@ -1,5 +1,5 @@ diff --git a/compiler/rustc_codegen_llvm/src/back/llvm_backend.rs b/compiler/rustc_codegen_llvm/src/back/llvm_backend.rs -index 8780af2f6..374c78070 100644 +index 8780af2f6..b5d721623 100644 --- a/compiler/rustc_codegen_llvm/src/back/llvm_backend.rs +++ b/compiler/rustc_codegen_llvm/src/back/llvm_backend.rs @@ -74,6 +74,39 @@ fn compile_codegen_unit( @@ -42,6 +42,24 @@ index 8780af2f6..374c78070 100644 } impl WriteBackendMethods for LlvmCodegenBackend { +@@ -191,7 +224,7 @@ fn init(&mut self, sess: &EarlySession) -> CodegenBackendInit { + + use crate::back::lto::enable_autodiff_settings; + if sess.opts.unstable_opts.autodiff.contains(&AutoDiff::Enable) { +- match llvm::EnzymeWrapper::get_or_init(&sess.opts.sysroot) { ++ match llvm::EnzymeWrapper::get_or_init(&(*sess.opts.read_sysroot())) { + Ok(_) => {} + Err(llvm::EnzymeLibraryError::NotFound { err }) => { + sess.dcx().emit_fatal(crate::diagnostics::AutoDiffComponentMissing { err }); +@@ -339,7 +372,7 @@ fn codegen_crate<'tcx>(&self, tcx: TyCtxt<'tcx>) -> Box { + if tcx.sess.opts.unstable_opts.offload.iter().any(|o| matches!(o, Offload::Device(_))) + || tcx.sess.opts.unstable_opts.offload.iter().any(|o| matches!(o, Offload::Host(_))) + { +- match llvm::RustOffloadWrapper::get_or_init(&tcx.sess.opts.sysroot) { ++ match llvm::RustOffloadWrapper::get_or_init(&(*tcx.sess.opts.read_sysroot())) { + Ok(_) => {} + Err(llvm::RustOffloadLibraryError::NotFound { err }) => { + tcx.sess diff --git a/compiler/rustc_codegen_llvm/src/base.rs b/compiler/rustc_codegen_llvm/src/base.rs index 77cb37ede..4bd2556f3 100644 --- a/compiler/rustc_codegen_llvm/src/base.rs @@ -331,7 +349,7 @@ index f576f29a1..bc9ce9580 100644 pub(crate) fn LLVMIsMultithreaded() -> Bool; diff --git a/compiler/rustc_codegen_ssa/src/base.rs b/compiler/rustc_codegen_ssa/src/base.rs -index f9dfd04fa..fea558316 100644 +index f9dfd04fa..d6f8447fe 100644 --- a/compiler/rustc_codegen_ssa/src/base.rs +++ b/compiler/rustc_codegen_ssa/src/base.rs @@ -1,4 +1,5 @@ @@ -340,6 +358,15 @@ index f9dfd04fa..fea558316 100644 use std::sync::Arc; use std::time::{Duration, Instant}; use std::{cmp, iter}; +@@ -820,7 +821,7 @@ pub fn codegen_crate< + // This likely is a temporary measure. Once we don't have to support the + // non-parallel compiler anymore, we can compile CGUs end-to-end in + // parallel and get rid of the complicated scheduling logic. +- let mut pre_compiled_cgus = if let Some(threads) = tcx.sess.opts.jobs.frontend { ++ let mut pre_compiled_cgus = if let Some(threads) = (*tcx.sess.opts.read_jobs()).frontend { + tcx.sess.time("compile_first_CGU_batch", || { + // Try to find one CGU to compile per thread. + let cgus: Vec<_> = cgu_reuse @@ -847,6 +848,7 @@ pub fn codegen_crate< FxHashMap::default() }; @@ -465,9 +492,18 @@ index e06ac36fd..1fea1d7c8 100644 + } } diff --git a/compiler/rustc_incremental/src/persist/save.rs b/compiler/rustc_incremental/src/persist/save.rs -index 46f47d6c8..69ea0af7a 100644 +index 46f47d6c8..dec36c5cd 100644 --- a/compiler/rustc_incremental/src/persist/save.rs +++ b/compiler/rustc_incremental/src/persist/save.rs +@@ -26,7 +26,7 @@ pub(crate) fn save_dep_graph(tcx: TyCtxt<'_>) { + debug!("save_dep_graph()"); + tcx.dep_graph.with_ignore(|| { + let sess = tcx.sess; +- if sess.opts.incremental.is_none() { ++ if (*sess.opts.read_incremental()).is_none() { + return; + } + // This is going to be deleted in finalize_session_directory, so let's not create it. @@ -41,6 +41,7 @@ pub(crate) fn save_dep_graph(tcx: TyCtxt<'_>) { sess.time("assert_dep_graph", || assert_dep_graph(tcx)); @@ -476,8 +512,17 @@ index 46f47d6c8..69ea0af7a 100644 par_join( move || { +@@ -96,7 +97,7 @@ pub fn save_work_product_index( + dep_graph: &DepGraph, + new_work_products: WorkProductMap, + ) { +- if sess.opts.incremental.is_none() { ++ if (*sess.opts.read_incremental()).is_none() { + return; + } + // This is going to be deleted in finalize_session_directory, so let's not create it diff --git a/compiler/rustc_metadata/src/rmeta/encoder.rs b/compiler/rustc_metadata/src/rmeta/encoder.rs -index 03493d0e1..0adb62f8e 100644 +index 03493d0e1..77ba6f59c 100644 --- a/compiler/rustc_metadata/src/rmeta/encoder.rs +++ b/compiler/rustc_metadata/src/rmeta/encoder.rs @@ -52,6 +52,9 @@ @@ -503,6 +548,15 @@ index 03493d0e1..0adb62f8e 100644 let unhashed = stat!("final", || { // Indexed by dependency `CrateNum`, matching the numbering `encode_crate_deps` uses. +@@ -1940,7 +1947,7 @@ fn encode_mir(&mut self) { + // save the query traffic. + if tcx.sess.opts.output_types.should_codegen() + && tcx.sess.opts.optimize != OptLevel::No +- && tcx.sess.opts.incremental.is_none() ++ && (*tcx.sess.opts.read_incremental()).is_none() + { + for &local_def_id in tcx.mir_keys(()) { + if let DefKind::AssocFn | DefKind::Fn = tcx.def_kind(local_def_id) { @@ -2561,6 +2568,10 @@ pub fn encode_metadata(tcx: TyCtxt<'_>, path: &Path, ref_path: Option<&Path>) { let hash = blob.expect("file already created").get_crate_hash(); tcx.untracked().local_crate_hash.set(hash).expect("local_crate_hash set twice"); @@ -514,6 +568,15 @@ index 03493d0e1..0adb62f8e 100644 // Generate the metadata stub manually, as that is a small file compared to full metadata. if let Some(ref_path) = ref_path { let _prof_timer = tcx.prof.verbose_generic_activity("generate_crate_metadata_stub"); +@@ -2579,7 +2590,7 @@ pub fn encode_metadata(tcx: TyCtxt<'_>, path: &Path, ref_path: Option<&Path>) { + return; + }; + +- if tcx.sess.opts.jobs.frontend.is_some() { ++ if (*tcx.sess.opts.read_jobs()).frontend.is_some() { + // Prefetch some queries used by metadata encoding. + // This is not necessary for correctness, but is only done for performance reasons. + // It can be removed if it turns out to cause trouble or be detrimental to performance. @@ -2638,6 +2649,15 @@ fn with_encode_metadata_header( tcx: TyCtxt<'_>, path: &Path, @@ -627,9 +690,18 @@ index 7bccc34db..71f2341b1 100644 /// This checks non-recursive runtime validity. hook validate_scalar_in_layout(scalar: crate::ty::ScalarInt, ty: Ty<'tcx>) -> bool; diff --git a/compiler/rustc_middle/src/query/on_disk_cache.rs b/compiler/rustc_middle/src/query/on_disk_cache.rs -index 956c59012..957ddcb20 100644 +index 956c59012..04a593b5d 100644 --- a/compiler/rustc_middle/src/query/on_disk_cache.rs +++ b/compiler/rustc_middle/src/query/on_disk_cache.rs +@@ -153,7 +153,7 @@ impl OnDiskCache { + /// The serialized cache has some basic integrity checks, if those checks indicate that the + /// on-disk data is corrupt, an error is returned. + pub fn new(sess: &Session, data: Mmap, start_pos: usize) -> Result { +- assert!(sess.opts.incremental.is_some()); ++ assert!((*sess.opts.read_incremental()).is_some()); + + let mut decoder = MemDecoder::new(&data, start_pos)?; + @@ -756,6 +756,51 @@ pub struct CacheEncoder<'tcx> { side_effects_index: Vec<(SerializedDepNodeIndex, AbsoluteBytePos)>, } diff --git a/rustc/artifacts.py b/rustc/artifacts.py index f8ebad7..aa0ae18 100644 --- a/rustc/artifacts.py +++ b/rustc/artifacts.py @@ -18,7 +18,7 @@ from collections import Counter from pathlib import Path -SESSION = re.compile(rb"\.[0-9a-z]{7}\.rcgu\.o") +SESSION = re.compile(rb"\.[0-9a-z]{7}(\.rcgu\.(?:o|dwo))") def ar_members(data): @@ -41,15 +41,15 @@ def ar_members(data): offset = int(name[1:]) name = names[offset:names.index(b"/\n", offset)] name = name.rstrip(b"/") - members[SESSION.sub(b".rcgu.o", name).decode("utf-8", "replace")] = body + members[SESSION.sub(rb"\1", name).decode("utf-8", "replace")] = body return members def normalized_rlib(path): out = {} for name, body in ar_members(Path(path).read_bytes()).items(): - if name == "lib.rmeta-link": - body = SESSION.sub(b".rcgu.o", body) + # The link metadata and, with split debuginfo, the objects name session-suffixed files. + body = SESSION.sub(rb"\1", body) out[name] = hashlib.sha256(body).hexdigest() return out @@ -81,7 +81,7 @@ def collect(stdout, target): if msg.get("executable"): f = msg["executable"] rel = str(Path(f).relative_to(target)) if f.startswith(str(target)) else f - found["exe"][rel] = hashlib.sha256(Path(f).read_bytes()).hexdigest() + found["exe"][rel] = hashlib.sha256(SESSION.sub(rb"\1", Path(f).read_bytes())).hexdigest() return found diff --git a/rustc/flag-campaign.sh b/rustc/flag-campaign.sh new file mode 100755 index 0000000..617d995 --- /dev/null +++ b/rustc/flag-campaign.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# A sequence of flag-transition walks on fixtures/sink that stops at the first new finding. +# +# rustc/flag-campaign.sh [--recheck] +# +# holds the tables and one walk directory per walk; rerunning resumes where it stopped. +# The compiler is /rustc (a symlink to a toolchain directory). When a walk pauses on a +# finding (its PAUSED file names the row), the campaign exits 3; patch rustc, point +# /rustc at the patched toolchain, and rerun with --recheck: the rows with findings run +# again first, then everything not yet walked. +# +# Needs PICT (PICT=..., default ~/mirth-work/tools/pict/pict) and flag-universe.py's results +# (FLAGS=..., default ~/mirth-work/flags). +set -u +D=$(cd "$1" && pwd); shift +here=$(cd "$(dirname "$0")" && pwd) +PICT=${PICT:-$HOME/mirth-work/tools/pict/pict} +FLAGS=${FLAGS:-$HOME/mirth-work/flags} +FIXTURE=${FIXTURE:-$here/../fixtures/sink} +cd "$D" + +model() { # name subset + [ -f "$1.txt" ] || python3 "$here/flag-model.py" "$FLAGS" "$2" "$1.txt" --transitions --cargo --allow-known > /dev/null +} +table() { # model strength seed + t="$1-t$2-r$3.tsv" + [ -f "$t" ] || "$PICT" "$1.txt" /o:$2 /r:$3 > "$t" 2> /dev/null + echo "$t" +} +walk() { # table edits seed [flag-walk options] + local tab=$1 edits=$2 seed=$3; shift 3 + w="walk-${tab%.tsv}-e$edits" + python3 "$here/flag-walk.py" --rustc "$D/rustc/bin/rustc" --fixture "$FIXTURE" --flags "$FLAGS" \ + --table "$tab" --work "$w" --workers 10 --edits "$edits" --seed "$seed" --pause-on-finding "$@" \ + > "$w.log" 2>&1 + echo "$(date +%T) $w $(tail -1 "$w.log")" | tee -a campaign.log + [ ! -f "$w/PAUSED" ] || exit 3 +} + +model all all +model untracked untracked +model tracked tracked +for s in 1 2 3; do + walk "$(table all 2 $s)" 0 $s "$@" + walk "$(table all 2 $s)" 2 $s "$@" +done +for s in 1 2; do + walk "$(table untracked 3 $s)" 2 $s "$@" +done +for s in 1 2 3 4 5 6; do + walk "$(table all 2 $((s + 10)))" 3 $s "$@" +done +echo "$(date +%T) DONE" | tee -a campaign.log diff --git a/rustc/flag-min.py b/rustc/flag-min.py new file mode 100644 index 0000000..f9c37a3 --- /dev/null +++ b/rustc/flag-min.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""Minimize the failures of a flag-walk.py run: for each distinct first error among rows +whose clean build with A failed, find the smallest set of the row's options that still gives +the same error (delta debugging, a clean build of the fixture per test). + + rustc/flag-min.py --rustc --fixture fixtures/sink --walk + [--per-error 1] [--jobs 4] + +Prints one line per error: the minimal options. Writes /minimized.json. +""" + +import argparse +import json +import os +import re +import shutil +import subprocess +import tempfile +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +p = argparse.ArgumentParser() +p.add_argument("--rustc", required=True) +p.add_argument("--fixture", required=True) +p.add_argument("--walk", required=True) +p.add_argument("--per-error", type=int, default=1) +p.add_argument("--jobs", type=int, default=4) +p.add_argument("--toolchain", default="nightly-2026-10-06") +p.add_argument("--target", default="x86_64-unknown-linux-gnu") +args = p.parse_args() + +FLAG_BASE = ("-Cunsafe-allow-abi-mismatch=sanitizer,sanitizer-cfi-normalize-integers," + "sanitizer-cfi-minimal-runtime,retpoline,retpoline-external-thunk," + "indirect-branch-cs-prefix,fixed-x18,reg-struct-return,regparm,branch-protection") +WALK = Path(args.walk) + + +def signature(line): + """An error line with paths, symbols, hashes and quoted names removed.""" + line = re.sub(r"_R\w+|/\S+|`[^`]*`|\b[0-9a-f]{16}\b", "…", line) + line = re.sub(r"\d+", "N", line) + return line[:120] + + +def first_error(log): + for l in log.splitlines(): + if l.startswith(("error", "rustc-LLVM ERROR", "LLVM ERROR")) and "could not compile" not in l: + return l + if "panicked at" in l: + return l + return "" + + +def minimize(flags, sig): + work = Path(tempfile.mkdtemp(dir=WALK)) + src, target = work / "s", work / "t" + shutil.copytree(Path(args.fixture), src, ignore=shutil.ignore_patterns("target", "edits", "edit")) + e = dict(os.environ, RUSTC=args.rustc, RUSTC_WRAPPER="", CARGO_INCREMENTAL="1", CARGO_TERM_COLOR="never") + + def fails(fl): + shutil.rmtree(target, ignore_errors=True) + e["RUSTFLAGS"] = " ".join([FLAG_BASE] + fl) + r = subprocess.run(["cargo", f"+{args.toolchain}", "build", "--workspace", "--offline", "-j", "4", + "--target", args.target, "--target-dir", str(target)], + cwd=src, env=e, capture_output=True, text=True) + return r.returncode != 0 and signature(first_error(r.stderr)) == sig + + if not fails(flags): + shutil.rmtree(work) + return None + n = 2 + while len(flags) >= 2: + chunk = max(1, len(flags) // n) + for i in range(0, len(flags), chunk): + rest = flags[:i] + flags[i + chunk:] + if fails(rest): + flags, n = rest, max(n - 1, 2) + break + else: + if chunk == 1: + break + n = min(len(flags), n * 2) + shutil.rmtree(work) + return flags + + +rows = [json.loads(l) for l in open(WALK / "results.jsonl")] +todo = {} +for r in rows: + if r["A_ok"]: + continue + err = (r.get("errors") or [r.get("error", "")])[0] + sig = signature(err) + todo.setdefault(sig, []) + if len(todo[sig]) < args.per_error: + todo[sig].append(r["A"]) + +jobs = [(sig, fl) for sig, fls in todo.items() for fl in fls] +with ThreadPoolExecutor(args.jobs) as ex: + found = list(ex.map(lambda j: (j[0], minimize(list(j[1]), j[0])), jobs)) +out = [{"error": sig, "minimal": fl} for sig, fl in found] +(WALK / "minimized.json").write_text(json.dumps(out, indent=1)) +for o in out: + print(o["minimal"], "->", o["error"]) diff --git a/rustc/flag-model.py b/rustc/flag-model.py new file mode 100644 index 0000000..13fbc8a --- /dev/null +++ b/rustc/flag-model.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +"""Write a PICT model of rustc's option universe from flag-universe.py's results. + + rustc/flag-model.py [--transitions] [--cargo] [--allow-known] + +One parameter per option. Its values are absence, the values rustc accepted alone, the +values it accepts once `-Cunsafe-allow-abi-mismatch` names every target modifier (FLAG_BASE, +passed on every row), and the values that need another option, with IF/THEN constraints for +those needs. Options that stop compilation early (help, parse-only, link-only) are left out. +A value whose need lies outside the subset is dropped. + +With --cargo the model is for building a Cargo workspace (flag-walk.py): it leaves out the +values in CARGO_DROP, which fail there for reasons of Cargo or this machine, not the options. + +Combinations that hit bugs already in docs/hunt.md are excluded, so walks look for new +ones; --allow-known keeps those that have a local stopgap (for a compiler with them). + +With --transitions every parameter appears twice, A_ before and B_ after, for covering the +changes between two sessions. + +Run it with PICT (github.com/microsoft/pict): `pict /o:2` gives a pairwise covering +array, `/o:3` three-way. +""" + +import json +import re +import sys + +FLAG_BASE = ("-Cunsafe-allow-abi-mismatch=sanitizer,sanitizer-cfi-normalize-integers," + "sanitizer-cfi-minimal-runtime,retpoline,retpoline-external-thunk," + "indirect-branch-cs-prefix,fixed-x18,reg-struct-return,regparm,branch-protection") +STOP = {"-Chelp", "-Zhelp", "-Zparse-crate-root-only", "-Zno-analysis", "-Zlink-only", + "-Zimplicit-sysroot-deps"} +CARGO_DROP = { + "-Zassert-incr-state": None, # fails whenever the cache state differs, by design + "-Zbuild-sdylib-interface": None, # Cargo's target probe fails + "-Zchecksum-hash-algorithm": ["md5", "sha1"], # Cargo cannot parse the dep info + "-Zdirect-access-external-data": None, # link fails + "-Zfunction-return": ["thunk-extern"], # link fails: no thunk + "-Zlink-native-libraries": None, # link fails + "-Zlint-llvm-ir": None, # aborts on known LLVM lint findings (rust-lang/rust#59793) + "-Zno-codegen": None, "-Zno-link": None, # later crates need the output + "-Zpanic-in-drop": None, # std is built with unwind + "-Zsanitizer": None, # no sanitizer runtimes in this sysroot: link fails + "-Zretpoline-external-thunk": None, # link fails: no thunk + "-Ztiny-const-eval-limit": None, # the fixture's const evaluation exceeds it + "-Cpanic": ["immediate-abort"], # core is built with unwind + "-Ccode-model": ["tiny"], # LLVM ERROR: not supported on x86_64 + "-Ztls-model": ["local-exec", "emulated"], # dylib cannot link + # the dylib cannot link (static, pie, ropi); rwpi: finding 12 + "-Crelocation-model": ["static", "pie", "ropi", "rwpi", "ropi-rwpi"], + "-Clto": None, # rejected for rlibs and dylibs; Cargo's profile applies it to final artifacts only + # LLVM's pass listing from codegen threads interleaves with rustc's lines on stderr; a split + # -Ztime-passes-format=json line then reaches Cargo as a bare JSON message. + "-Zprint-llvm-passes": ["yes"], + "-Ztime-passes-format": ["json"], +} +# Constraints that only a real workspace shows: a binary, a dylib, Cargo's own flags. +CARGO_NEEDS = [ + ("-Cprefer-dynamic", "yes", ["-Cpanic"], '[Cpanic] <> "abort"'), # libstd.so has panic_unwind + ("-Cprefer-dynamic", "yes", ["-Clto"], '[Clto] IN {"absent","no","off"}'), + # Findings 13 and 14 (in LLVM, not patched): retpolines with the machine outliner, or with + # the large code model. + ("-Zretpoline", "yes", ["-Ccode-model"], '[Ccode_model] <> "large"'), + # Not a bug: -Zmir-opt-bisect-limit counts pass runs across the session, so with a + # parallel frontend which bodies stay under the limit depends on thread timing. + ("-Zthreads", "4", ["-Zmir-opt-bisect-limit"], '[Zmir_opt_bisect_limit] = "absent"'), +] +# Bugs in docs/hunt.md that have a local stopgap: excluded unless --allow-known (for a +# compiler with the stopgaps). +KNOWN_NEEDS = [ + # Finding 9: without the default passes, local ThinLTO leaves undefined hidden symbols. + ("-Cno-prepopulate-passes", "present", ["-Zthinlto", "-Copt-level"], + '[Zthinlto] <> "yes" AND [Copt_level] IN {"absent","0"}'), +] +# Values left out of every model: LLVM's machine outliner crashes in many combinations +# (finding 13), which buries everything else. +DROP = {"-Cllvm-args": ["-enable-machine-outliner"]} +# Rejected alone; accepted with FLAG_BASE or with the needs below. +EXTRA = {"-Zindirect-branch-cs-prefix": ["yes"], "-Zretpoline-external-thunk": ["yes"], + "-Zretpoline": ["yes"], + "-Zsanitizer": ["dataflow", "memory", "safestack", "thread", "cfi", "kcfi"], + "-Cforce-frame-pointers": ["non-leaf"], "-Cpanic": ["immediate-abort"], + "-Zdump-dep-graph": ["yes"], "-Zsanitizer-cfi-canonical-jump-tables": ["no"], + "-Zsanitizer-cfi-diag": ["yes"], "-Zsanitizer-cfi-generalize-pointers": ["yes"], + "-Zsanitizer-cfi-minimal-runtime": ["yes"], "-Zsanitizer-cfi-normalize-integers": ["yes"], + "-Zsanitizer-cfi-recover": ["yes"], "-Zsanitizer-kcfi-arity": ["yes"], + "-Zsplit-lto-unit": ["yes"], "-Zvirtual-function-elimination": ["yes"]} +LTO_ON = '{"yes","on","thin","fat"}' +LTO_FAT = '{"yes","on","fat"}' +# (option, value, options needed, PICT condition) +NEEDS = [ + ("-Cembed-bitcode", "no", ["-Clto"], '[Clto] IN {"absent","no","off"}'), + ("-Zsplit-lto-unit", "yes", ["-Clto"], "[Clto] IN " + LTO_ON), + ("-Zvirtual-function-elimination", "yes", ["-Clto"], "[Clto] IN " + LTO_FAT), + ("-Zsanitizer", "cfi", ["-Clto", "-Ccodegen-units"], "[Clto] IN " + LTO_FAT + ' AND [Ccodegen_units] = "1"'), + ("-Zsanitizer", "kcfi", ["-Cpanic"], '[Cpanic] = "abort"'), + ("-Cforce-frame-pointers", "non-leaf", ["-Zunstable-options"], '[Zunstable_options] = "present"'), + ("-Cpanic", "immediate-abort", ["-Zunstable-options"], '[Zunstable_options] = "present"'), + ("-Zdump-dep-graph", "yes", ["-Zquery-dep-graph"], '[Zquery_dep_graph] = "yes"'), + ("-Zsanitizer-cfi-diag", "yes", ["-Zsanitizer"], '[Zsanitizer] = "cfi"'), + ("-Zsanitizer-cfi-recover", "yes", ["-Zsanitizer"], '[Zsanitizer] = "cfi"'), + ("-Zsanitizer-cfi-minimal-runtime", "yes", ["-Zsanitizer"], '[Zsanitizer] = "cfi"'), + ("-Zsanitizer-cfi-canonical-jump-tables", "no", ["-Zsanitizer"], '[Zsanitizer] = "cfi"'), + ("-Zsanitizer-cfi-generalize-pointers", "yes", ["-Zsanitizer"], '[Zsanitizer] IN {"cfi","kcfi"}'), + ("-Zsanitizer-cfi-normalize-integers", "yes", ["-Zsanitizer"], '[Zsanitizer] IN {"cfi","kcfi"}'), + ("-Zsanitizer-kcfi-arity", "yes", ["-Zsanitizer"], '[Zsanitizer] = "kcfi"'), + ("-Zsanitizer-cfi-minimal-runtime", "yes", ["-Zsanitizer-cfi-recover", "-Zsanitizer-cfi-diag"], + '([Zsanitizer_cfi_recover] = "yes" OR [Zsanitizer_cfi_diag] = "yes")'), +] + + +def pname(opt): + return re.sub(r"[^A-Za-z0-9]", "_", opt.lstrip("-")) + + +def main(): + work, subset, out = sys.argv[1:4] + transitions = "--transitions" in sys.argv[4:] + cargo = "--cargo" in sys.argv[4:] + known = "--allow-known" not in sys.argv[4:] + opts = {o["flag"] + o["name"]: o for o in json.load(open(work + "/options.json"))} + domains = {} + for s in json.load(open(work + "/singles.json")): + if s["ok"] and s["option"] not in STOP: + domains.setdefault(s["option"], []).append("present" if s["value"] is None else s["value"]) + for k, vs in EXTRA.items(): + for v in vs: + if v not in domains.setdefault(k, []): + domains[k].append(v) + for k, vs in DROP.items(): + domains[k] = [v for v in domains.get(k, []) if v not in vs] + if cargo: + for k, vs in CARGO_DROP.items(): + domains[k] = [] if vs is None else [v for v in domains.get(k, []) if v not in vs] + domains = {k: v for k, v in domains.items() if v} + + keep = [k for k in sorted(domains) + if subset == "all" or (subset == "untracked") == (opts[k]["tracking"] == "UNTRACKED")] + cons = [] + for k, v, needs, cond in NEEDS + (CARGO_NEEDS if cargo else []) + (KNOWN_NEEDS if known else []): + if k not in keep: + continue + if all(n in keep for n in needs): + cons.append(f'IF [{pname(k)}] = "{v}" THEN {cond};') + elif v in domains[k]: + domains[k].remove(v) + + params = [f"{pname(k)}: {', '.join(['absent'] + [v.replace(',', ';') for v in domains[k]])}" for k in keep] + if transitions: + params = [f"{t}_{p}" for t in "AB" for p in params] + cons = [re.sub(r"\[(\w+)\]", lambda m: f"[{t}_{m.group(1)}]", c) for t in "AB" for c in cons] + if known and "-Zprint-type-sizes" in keep: + # Finding 11 in docs/hunt.md: the rebuild ICEs once -Zprint-type-sizes is dropped. + cons.append('IF [A_Zprint_type_sizes] = "yes" THEN [B_Zprint_type_sizes] = "yes";') + open(out, "w").write("\n".join(params) + "\n\n" + "\n".join(cons) + "\n") + print(f"{len(params)} parameters, {len(cons)} constraints") + + +if __name__ == "__main__": + main() diff --git a/rustc/flag-rows.py b/rustc/flag-rows.py new file mode 100644 index 0000000..196bdcc --- /dev/null +++ b/rustc/flag-rows.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +"""Compile a trivial crate once per row of a PICT table made from flag-model.py's model, and +count the rows rustc rejects, grouped by first error. + + rustc/flag-rows.py [extra rustc args, e.g. --emit=metadata] + +Tables from a --transitions model are not supported. +""" + +import csv +import importlib.util +import json +import subprocess +import sys +import tempfile +from collections import Counter +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +spec = importlib.util.spec_from_file_location("flag_model", Path(__file__).with_name("flag-model.py")) +flag_model = importlib.util.module_from_spec(spec) +spec.loader.exec_module(flag_model) + +work, tsv, rustc = sys.argv[1:4] +extra = sys.argv[4:] +opts = {flag_model.pname(o["flag"] + o["name"]): o for o in json.load(open(work + "/options.json"))} +rows = list(csv.DictReader(open(tsv), delimiter="\t")) + + +def argv(row): + out = [flag_model.FLAG_BASE] + for k, v in row.items(): + if v != "absent": + f = opts[k]["flag"] + opts[k]["name"] + out.append(f if v == "present" else f + "=" + v.replace(";", ",")) + return out + + +def run(row): + with tempfile.TemporaryDirectory(dir=work) as d: + Path(d, "lib.rs").write_text("pub fn f(x: u32) -> u32 { x.wrapping_mul(3) }\n") + a = argv(row) + p = subprocess.run([rustc, "--edition", "2021", "--crate-type", "lib", *extra, "-o", d + "/out", + *a, d + "/lib.rs"], capture_output=True, text=True, cwd=d, timeout=300) + err = next((l for l in p.stderr.splitlines() if l.startswith("error")), "") + return p.returncode == 0, err, len(a) - 1 + + +res = list(ThreadPoolExecutor(10).map(run, rows)) +bad = [e for ok, e, _ in res if not ok] +print(f"{len(rows)} rows, {len(bad)} rejected, {sum(n for *_, n in res) / len(res):.0f} options per row on average") +for e, c in Counter(e[:110] for e in bad).most_common(): + print(c, e) diff --git a/rustc/flag-universe.py b/rustc/flag-universe.py new file mode 100644 index 0000000..7f9bc19 --- /dev/null +++ b/rustc/flag-universe.py @@ -0,0 +1,251 @@ +#!/usr/bin/env python3 +"""Enumerate rustc's -C and -Z options, find which values and pairs of values it accepts, +and size covering arrays over them. + + rustc/flag-universe.py --rustc --source --work [--jobs N] + +Each option's domain is its absence plus the values worth trying: `yes`/`no` for a boolean, +present for an option without a value, the values its parser's description lists for an +enumerated one, two samples for a number. Options taking free-form strings, paths or lists +are counted but left out. Every value is tried alone on a trivial crate (`--emit=metadata`, +so only option checking and a tiny compilation run), then every pair of accepted values of +different options. A pair is "rejected" when rustc fails with the pair but accepts each value +alone. + +Writes /options.json (domains), /singles.json, /pairs.json, and prints +the sizes of pairwise covering arrays built greedily over the accepted domains. +""" + +import argparse +import itertools +import json +import os +import random +import re +import subprocess +import tempfile +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +p = argparse.ArgumentParser() +p.add_argument("--rustc", required=True) +p.add_argument("--source", required=True, help="a rust checkout, for compiler/rustc_session/src/options.rs") +p.add_argument("--work", required=True) +p.add_argument("--jobs", type=int, default=8) +p.add_argument("--skip-pairs", action="store_true") +args = p.parse_args() + +work = Path(args.work) +work.mkdir(parents=True, exist_ok=True) +src = (Path(args.source) / "compiler/rustc_session/src/options.rs").read_text() + +# The parsers' descriptions, joined across lines. +descs = {} +for m in re.finditer(r'pub\(crate\) const (parse_\w+): &str =\s*((?:"(?:[^"\\]|\\.)*"\s*)+|parse_\w+|[^;]+);', src): + descs[m.group(1)] = m.group(2) +for k, v in list(descs.items()): + if re.fullmatch(r"parse_\w+", v.strip()): + descs[k] = descs.get(v.strip(), "") + +BOOL = {"parse_bool", "parse_opt_bool"} +NO_VALUE = {"parse_no_value"} +NUMBER = {"parse_number", "parse_opt_number"} +FREE = {"parse_string", "parse_opt_string", "parse_string_push", "parse_opt_pathbuf", "parse_list", + "parse_comma_list", "parse_opt_comma_list", "parse_ignore", "parse_target_feature", + "parse_list_with_polarity", "parse_llvm_module_flag", "parse_patchable_function_entry", + "parse_autodiff", "parse_offload", "parse_allow_partial_mitigations", + "parse_deny_partial_mitigations", "parse_rust_version", "parse_unpretty", + "parse_passes", "parse_branch_protection", "parse_instrument_xray", + "parse_linker_features", "parse_link_self_contained", "parse_align", + "parse_location_detail", "parse_coverage_options", "parse_codegen_retag_options"} + + +# Values for options whose parser takes a string or whose description lists no values, picked +# by hand (`rustc --print code-models` etc. for the enumerations). +SAMPLES = { + "-Copt-level": ["0", "1", "2", "3", "s", "z"], + "-Ccode-model": ["tiny", "small", "kernel", "medium", "large"], + "-Crelocation-model": ["static", "pic", "pie", "dynamic-no-pic", "ropi", "rwpi", "ropi-rwpi", "default"], + "-Ztls-model": ["global-dynamic", "local-dynamic", "initial-exec", "local-exec", "emulated"], + "-Ctarget-cpu": ["generic", "native", "x86-64-v2", "x86-64-v3", "x86-64-v4"], + "-Ctarget-feature": ["+avx2", "+avx512f", "-sse4.2", "+crt-static"], + "-Ztune-cpu": ["generic", "znver4"], + "-Zthreads": ["1", "4"], + "-Zlocation-detail": ["none", "file", "line,column"], + "-Zmin-function-alignment": ["16", "64"], + "-Zpatchable-function-entry": ["4", "4,2"], + "-Zmir-enable-passes": ["+Inline", "-GVN", "+DeadStoreElimination-final"], + "-Zremap-cwd-prefix": ["/remapped"], + "-Zsimulate-remapped-rust-src-base": ["/rustc/simulated"], + "-Zhint-msrv": ["1.60.0"], + "-Cmetadata": ["mirth"], + "-Zinstrument-xray": ["always", "never"], + "-Cllvm-args": ["-unroll-threshold=0", "-enable-machine-outliner"], + "-Zcrate-attr": ["allow(unused)"], +} + + +def enum_values(parser): + vals = re.findall(r"`([^`]+)`", descs.get(parser, "")) + out = [] + for v in vals: + if v in out or " " in v or "<" in v or "=" in v: + continue + out.append(v) + return out + + +options = [] +for flag, grp in (("-C", "CodegenOptions"), ("-Z", "UnstableOptions")): + i = src.index("options! {\n " + grp) + j = src.index("\n}", i) + for line in src[i:j].splitlines(): + m = re.match(r"^\s*(\w+): .*?, (parse_\w+), \[(\w+)", line) + if not m: + continue + name, parser, tracking = m.group(1).replace("_", "-"), m.group(2), m.group(3) + if parser in BOOL: + values = ["yes", "no"] + elif parser in NO_VALUE: + values = [None] + elif parser in NUMBER: + values = ["1", "16"] + elif parser in FREE: + values = [] + else: + values = enum_values(parser) + values = SAMPLES.get(flag + name.replace("_", "-"), values) + options.append({"flag": flag, "name": name, "parser": parser, "tracking": tracking, + "values": values, "free": not values}) + + +def arg(o, v): + return f"{o['flag']}{o['name']}" if v is None else f"{o['flag']}{o['name']}={v}" + + +def run(argv): + with tempfile.TemporaryDirectory(dir=work) as d: + lib = Path(d) / "lib.rs" + lib.write_text("pub fn f(x: u32) -> u32 { x.wrapping_mul(3) }\n") + try: + r = subprocess.run([args.rustc, "--edition", "2021", "--crate-type", "lib", "--emit=metadata", + "-o", str(Path(d) / "out.rmeta"), *argv, str(lib)], + capture_output=True, text=True, timeout=60, cwd=d) + err = r.stderr + first = next((l for l in err.splitlines() if l.startswith(("error", "warning"))), "") + return {"ok": r.returncode == 0, "warn": "warning" in err, "msg": first[:200]} + except subprocess.TimeoutExpired: + return {"ok": False, "warn": False, "msg": "timeout"} + + +(work / "options.json").write_text(json.dumps(options, indent=1)) +walkable = [o for o in options if not o["free"]] +print(f"{len(options)} options: {len(walkable)} with enumerable values, " + f"{len(options) - len(walkable)} free-form") + +singles_path = work / "singles.json" +singles = json.loads(singles_path.read_text()) if singles_path.exists() else [] +# Try the values not tried yet (all of them on a first run). +done = {s["arg"] for s in singles} +jobs = [(o, v) for o in walkable for v in o["values"] if arg(o, v) not in done] +if jobs: + with ThreadPoolExecutor(args.jobs) as ex: + results = list(ex.map(lambda ov: run([arg(*ov)]), jobs)) + singles += [{"arg": arg(o, v), "option": o["flag"] + o["name"], "value": v, **r} + for (o, v), r in zip(jobs, results)] + singles_path.write_text(json.dumps(singles, indent=1)) +accepted = {} +for s in singles: + if s["ok"]: + accepted.setdefault(s["option"], []).append(s["arg"]) +print(f"{len(singles)} single values tried, {sum(map(len, accepted.values()))} accepted " + f"({len(accepted)} options with at least one)") + +rejected_pairs = set() +if not args.skip_pairs: + pairs_path = work / "pairs.json" + if pairs_path.exists(): + pairs = json.loads(pairs_path.read_text()) + else: + opts = sorted(accepted) + jobs = [(a, b) for x, y in itertools.combinations(opts, 2) for a in accepted[x] for b in accepted[y]] + # A value rejected alone may need another option: try it with every accepted value + # of every other option. + for r in (x for x in singles if not x["ok"]): + jobs += [(r["arg"], b) for y in opts if y != r["option"] for b in accepted[y]] + print(f"{len(jobs)} pairs to try", flush=True) + with ThreadPoolExecutor(args.jobs) as ex: + results = list(ex.map(lambda ab: run(list(ab)), jobs)) + pairs = [{"a": a, "b": b, **r} for (a, b), r in zip(jobs, results)] + pairs_path.write_text(json.dumps(pairs, indent=1)) + alone = {x["arg"] for x in singles if x["ok"]} + rejected_pairs = {(x["a"], x["b"]) for x in pairs if not x["ok"] and x["a"] in alone and x["b"] in alone} + requires = {} + for x in pairs: + if x["ok"] and x["a"] not in alone: + requires.setdefault(x["a"], []).append(x["b"]) + print(f"{len(pairs)} pairs tried; {len(rejected_pairs)} pairs of values accepted alone " + f"are rejected together; {len(requires)} values rejected alone are accepted with " + f"another option") + (work / "requires.json").write_text(json.dumps(requires, indent=1)) + + +def covering_array(domains, forbidden, seed=0): + """A pairwise covering array, built greedily: each row is chosen among random + candidates to cover the most uncovered pairs. Values are indices into each domain; index + 0 is the option's absence.""" + rng = random.Random(seed) + names = list(domains) + n = len(names) + uncovered = set() + for i, j in itertools.combinations(range(n), 2): + for a in range(len(domains[names[i]])): + for b in range(len(domains[names[j]])): + if (domains[names[i]][a], domains[names[j]][b]) not in forbidden: + uncovered.add((i, a, j, b)) + rows = [] + while uncovered: + best, best_gain = None, -1 + target = next(iter(uncovered)) + for _ in range(30): + row = [rng.randrange(len(domains[k])) for k in names] + row[target[0]], row[target[2]] = target[1], target[3] + # repair forbidden pairs by falling back to absence + for i, j in itertools.combinations(range(n), 2): + if (domains[names[i]][row[i]], domains[names[j]][row[j]]) in forbidden: + if j not in (target[0], target[2]): + row[j] = 0 + elif i not in (target[0], target[2]): + row[i] = 0 + gain = sum(1 for i, j in itertools.combinations(range(n), 2) if (i, row[i], j, row[j]) in uncovered) + if gain > best_gain: + best, best_gain = row, gain + rows.append(best) + for i, j in itertools.combinations(range(n), 2): + uncovered.discard((i, best[i], j, best[j])) + return rows + + +def summarize(label, opts): + domains = {o: [None] + accepted[o] for o in opts} + sizes = sorted((len(v) for v in domains.values()), reverse=True) + total = 1 + for s in sizes: + total *= s + lower = sizes[0] * sizes[1] if len(sizes) > 1 else sizes[0] + forbidden = {(a, b) for a, b in rejected_pairs} | {(b, a) for a, b in rejected_pairs} + rows = covering_array(domains, forbidden) + print(f"{label}: {len(opts)} options, all combinations {total:.3e}, " + f"pairwise covering array {len(rows)} rows (lower bound {lower})") + return rows + + +byopt = {o["flag"] + o["name"]: o for o in options} +results = {} +for label, pred in (("untracked", lambda o: o["tracking"] == "UNTRACKED"), + ("tracked", lambda o: o["tracking"] != "UNTRACKED"), + ("all", lambda o: True)): + opts = sorted(k for k in accepted if pred(byopt[k])) + if len(opts) > 1: + results[label] = summarize(label, opts) +(work / "covering.json").write_text(json.dumps({k: len(v) for k, v in results.items()})) diff --git a/rustc/flag-walk.py b/rustc/flag-walk.py new file mode 100644 index 0000000..25df97b --- /dev/null +++ b/rustc/flag-walk.py @@ -0,0 +1,272 @@ +#!/usr/bin/env python3 +"""Walk option transitions on a fixture: for each row of a PICT table made from a +`flag-model.py --transitions` model, build the fixture clean with the A_ options, rebuild it +incrementally with the B_ options, build it clean with the B_ options, and compare the +rebuild with the clean build (metadata, object code, binary, diagnostics, the binary's +output), as fuzz.py does after an edit. + + rustc/flag-walk.py --rustc --fixture fixtures/sink --flags + --table rows.tsv --work [--workers 8] [--rows a:b] + +Rows change options only; the source is not edited. + +The options go in RUSTFLAGS with `--target` set, so they apply to the fixture's crates but +not to its build scripts and proc macros. RUSTC_VERIFY_REUSE and RUSTC_REPORT_UNTRACKED are +set, for a compiler with mirth's local patches. + +Writes /results.jsonl (one line per row) and /findings// for each row whose +rebuild differs from the clean build, or which crashed. + +To stay at the frontier: with --pause-on-finding the walk stops taking rows at the first +finding not marked known and writes /PAUSED. Patch the compiler, then run the same +command with --rustc --recheck: the rows with findings run again first, then the +rows not yet walked. A rerun never repeats rows that are done. +""" + +import argparse +import csv +import difflib +import importlib.util +import json +import os +import random +import shutil +import subprocess +import sys +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent)) +import artifacts # noqa: E402 +import mutations # noqa: E402 + +spec = importlib.util.spec_from_file_location("flag_model", Path(__file__).with_name("flag-model.py")) +flag_model = importlib.util.module_from_spec(spec) +spec.loader.exec_module(flag_model) + +p = argparse.ArgumentParser() +p.add_argument("--rustc", required=True) +p.add_argument("--fixture", required=True) +p.add_argument("--flags", required=True, help="flag-universe.py's work directory") +p.add_argument("--table", required=True) +p.add_argument("--work", required=True) +p.add_argument("--workers", type=int, default=8) +p.add_argument("--toolchain", default="nightly-2026-10-06") +p.add_argument("--target", default="x86_64-unknown-linux-gnu") +p.add_argument("--timeout", type=int, default=600) +p.add_argument("--rows", default="", help="a:b, a slice of the table") +p.add_argument("--edits", type=int, default=0, help="random source edits between A and B (fuzz.py's)") +p.add_argument("--seed", type=int, default=0) +p.add_argument("--pause-on-finding", action="store_true", + help="stop taking rows at the first finding not marked known; rerun to resume") +p.add_argument("--recheck", action="store_true", + help="on resume, run the rows that had findings again first (after patching rustc)") +p.add_argument("--p5-builds", type=int, default=12, help="clean rebuilds before a difference counts as reuse") +args = p.parse_args() + +FIXTURE = Path(args.fixture).resolve() +WORK = Path(args.work).resolve() +opts = {flag_model.pname(o["flag"] + o["name"]): o for o in json.load(open(Path(args.flags) / "options.json"))} + + +def flags(row, side): + """RUSTFLAGS for one side of a row. Cargo passes `-Cembed-bitcode=no` unless its profile + asks for LTO, and the profile's LTO reaches only the final artifacts, so `-Clto` goes in + RUSTFLAGS with `-Cembed-bitcode=yes` after Cargo's flag.""" + out = [flag_model.FLAG_BASE] + for k, v in row.items(): + if k.startswith(side + "_") and v != "absent": + o = opts[k[2:]] + f = o["flag"] + o["name"] + out.append(f if v == "present" else f + "=" + v.replace(";", ",")) + if row.get(side + "_Clto") in ("yes", "on", "thin", "fat") and row.get(side + "_Cembed_bitcode") == "absent": + out.append("-Cembed-bitcode=yes") + return out + + +def build(src, target, rustflags): + e = dict(os.environ) + e.update(RUSTC=args.rustc, RUSTC_WRAPPER="", CARGO_INCREMENTAL="1", CARGO_TERM_COLOR="never", + RUSTFLAGS=" ".join(rustflags), RUSTC_VERIFY_REUSE="1", RUSTC_REPORT_UNTRACKED="1") + try: + r = subprocess.run(["cargo", f"+{args.toolchain}", "build", "--workspace", "--offline", "-j", "4", + "--target", args.target, "--target-dir", str(target), + "--message-format=json-render-diagnostics"], + cwd=src, env=e, capture_output=True, text=True, timeout=args.timeout) + rc, out, log = r.returncode, r.stdout, r.stderr + except subprocess.TimeoutExpired as t: + rc, out, log = -1, t.stdout or "", (t.stderr or "") + f"\nkilled after {args.timeout}s" + out = out.decode() if isinstance(out, bytes) else out + log = log.decode() if isinstance(log, bytes) else log + exe = None + for line in out.splitlines(): + try: + msg = json.loads(line) + except ValueError: + continue + if msg.get("reason") == "compiler-artifact" and msg.get("executable") and msg["target"]["name"] == FIXTURE.name: + exe = msg["executable"] + return {"ok": rc == 0, "log": log, + "ice": "internal compiler error" in log or "the compiler unexpectedly panicked" in log + or "rustc interrupted by SIG" in log, + "reuse": sorted({l.split(":", 1)[1].strip()[:200] for l in log.splitlines() + if l.startswith("rustc-verify-reuse:")}), + "untracked": sorted({l.strip() for l in log.splitlines() if l.startswith("rustc-untracked-read:")}), + "error": next((l for l in log.splitlines() if l.startswith("error")), ""), + "errors": [l[:300] for l in log.splitlines() if l.startswith(("error", "rustc-LLVM ERROR", "LLVM ERROR")) + and "could not compile" not in l][:6], + "art": artifacts.collect(out, target) if rc == 0 else None, "exe": exe} + + +def run_exe(exe): + if not exe: + return None + try: + r = subprocess.run([exe], capture_output=True, text=True, timeout=30) + return [r.returncode, r.stdout[-2000:]] + except subprocess.TimeoutExpired: + return ["timeout", ""] + + +def edit(src, rng, n): + """Apply n random edits to the fixture's sources; returns the unified diff.""" + diff = "" + for k in range(n): + paths = sorted(p for p in src.rglob("*.rs") if "target" not in p.parts) + for _ in range(20): + path = rng.choice(paths) + fn = rng.choices([e for e, _ in mutations.EDITS], weights=[w for _, w in mutations.EDITS])[0] + if fn in (mutations.str_literal, mutations.int_literal) and path.name == "build.rs": + continue # as in fuzz.py: stale OUT_DIR files, or a build script that loops + old = path.read_text() + new = fn(old, rng, k) + if new is not None and new != old: + path.write_text(new) + rel = str(path.relative_to(src)) + diff += "".join(difflib.unified_diff(old.splitlines(True), new.splitlines(True), + "a/" + rel, "b/" + rel)) + break + return diff + + +def walk(i_row): + i, row = i_row + if (WORK / "PAUSED").exists() or (WORK / "STOP").exists(): + return None + home = WORK / f"r{i}" + if home.exists(): + shutil.rmtree(home) + src, target, inc_target = home / "src", home / "target", home / "target-inc" + shutil.copytree(FIXTURE, src, ignore=shutil.ignore_patterns("target", "edits", "edit")) + a, b = flags(row, "A"), flags(row, "B") + res = {"row": i, "A": a[1:], "B": b[1:]} + first = build(src, target, a) + res["A_ok"] = first["ok"] + findings = [] + if first["ice"]: + findings.append("ICE in clean A") + if first["ok"]: + res["diff"] = edit(src, random.Random(args.seed * 1_000_003 + i), args.edits) + inc = build(src, target, b) + target.rename(inc_target) + clean = build(src, target, b) + res.update(inc_ok=inc["ok"], clean_ok=clean["ok"], reuse=inc["reuse"], untracked=inc["untracked"]) + if inc["ice"]: + findings.append("ICE in rebuild B") + if clean["ice"]: + findings.append("ICE in clean B") + if inc["ok"] != clean["ok"]: + findings.append("split: rebuild " + ("ok" if inc["ok"] else "failed") + + ", clean " + ("ok" if clean["ok"] else "failed")) + if inc["ok"] and clean["ok"]: + diff = artifacts.compare(inc["art"], clean["art"]) + if row.get("B_Csplit_debuginfo") in ("packed", "unpacked"): + # Objects and binary name .dwo files by session (DW_AT_GNU_dwo_name, and the + # dwo_id hashed from it), so two clean builds differ too. + diff.pop("rlib", None) + diff.pop("exe", None) + ra = run_exe(inc["exe"].replace(str(target), str(inc_target)) if inc["exe"] else None) + rb = run_exe(clean["exe"]) + if ra != rb: + findings.append(f"run: {ra} vs {rb}") + if diff: + # Clean builds may differ among themselves (P5), sometimes only one time in + # five: build clean again up to --p5-builds times before calling it reuse. + p5 = {} + for _ in range(args.p5_builds): + shutil.rmtree(target) + again = build(src, target, b) + if again["ok"]: + p5 = artifacts.compare(clean["art"], again["art"]) + if p5 or not again["ok"]: + break + kind = "P5 " if p5 else "" + findings += [f"{kind}{k}: {v[:5]}" for k, v in diff.items()] + res["error"] = inc["error"] or clean["error"] + res["errors"] = inc["errors"] or clean["errors"] + if findings: + d = WORK / "findings" / f"r{i}" + d.mkdir(parents=True, exist_ok=True) + (d / "row.json").write_text(json.dumps({**res, "findings": findings}, indent=1)) + if res.get("diff"): + (d / "edit.diff").write_text(res["diff"]) + (d / "inc.log").write_text(inc["log"][-20000:]) + (d / "clean.log").write_text(clean["log"][-20000:]) + else: + res["error"] = first["error"] + res["errors"] = first["errors"] + if findings: + d = WORK / "findings" / f"r{i}" + d.mkdir(parents=True, exist_ok=True) + (d / "row.json").write_text(json.dumps({**res, "findings": findings}, indent=1)) + (d / "a.log").write_text(first["log"][-20000:]) + res["findings"] = findings + res["rustc"] = args.rustc + new = [f for f in findings if not f.startswith("known")] + if new and args.pause_on_finding: + (WORK / "PAUSED").write_text(json.dumps({"row": i, "findings": new}, indent=1)) + shutil.rmtree(home) + with open(WORK / "results.jsonl", "a") as f: + f.write(json.dumps(res) + "\n") + print(f"row {i}: A {'ok' if res['A_ok'] else 'failed'}" + + (f", rebuild {'ok' if res.get('inc_ok') else 'failed'}" if res["A_ok"] else "") + + (f"; {findings}" if findings else "") + (f"; {res['error'][:100]}" if res.get("error") else ""), + flush=True) + return res + + +def latest(): + """The last result of each row, from /results.jsonl.""" + out = {} + path = WORK / "results.jsonl" + if path.exists(): + for line in path.read_text().splitlines(): + r = json.loads(line) + out[r["row"]] = r + return out + + +if __name__ == "__main__": + WORK.mkdir(parents=True, exist_ok=True) + (WORK / "PAUSED").unlink(missing_ok=True) + rows = list(csv.DictReader(open(args.table), delimiter="\t")) + idx = list(range(len(rows))) + if args.rows: + lo, hi = (int(x) if x else None for x in args.rows.split(":")) + idx = idx[lo:hi] + # Resume: rows with a result are done, except, with --recheck, those with findings, which + # run first. + done = latest() + again = [i for i in idx if i in done and done[i]["findings"]] if args.recheck else [] + idx = again + [i for i in idx if i not in done] + if again: + print(f"rechecking rows {again}", flush=True) + with ThreadPoolExecutor(args.workers) as ex: + list(ex.map(walk, [(i, rows[i]) for i in idx])) + results = list(latest().values()) + summary = {"rows": len(rows), "done": len(results), "A ok": sum(r["A_ok"] for r in results), + "compared": sum(bool(r.get("inc_ok") and r.get("clean_ok")) for r in results), + "findings": sum(bool(r["findings"]) for r in results)} + if (WORK / "PAUSED").exists(): + summary["paused"] = json.loads((WORK / "PAUSED").read_text()) + print(json.dumps(summary)) diff --git a/rustc/fuzz.py b/rustc/fuzz.py index 332058c..1e6bde8 100755 --- a/rustc/fuzz.py +++ b/rustc/fuzz.py @@ -27,7 +27,9 @@ rustc/fuzz.py --rustc --fixture fixtures/sink --work [--workers 8] [--edits N] -Stop it early by creating /STOP. Progress is in /stats.json. +Stop it early by creating /STOP. Progress is in /stats.json. With +--pause-on-finding all workers stop at the first finding (/PAUSED says which); patch +rustc and run again with the patched compiler. """ import argparse @@ -65,206 +67,16 @@ help="cargo check instead of cargo build: metadata only, no code or binaries") p.add_argument("--no-verify-reuse", action="store_true", help="do not set RUSTC_VERIFY_REUSE (needs a compiler with docs/hunt/verify-reuse.patch)") +p.add_argument("--pause-on-finding", action="store_true", + help="stop all workers at the first finding (writes /PAUSED); patch rustc and " + "run again to resume") args = p.parse_args() WORK = Path(args.work).resolve() FIXTURE = Path(args.fixture).resolve() BIN = FIXTURE.name -# ---------------------------------------------------------------- edits - - -def blocks(text): - """Top-level items: blank-line separated, continuation blocks merged.""" - out = [] - for b in text.split("\n\n"): - if out and (b[:1].isspace() or b.startswith("}") or b.startswith("where")): - out[-1] += "\n\n" + b - else: - out.append(b) - return out - - -def edit_lines(text, rng, f): - lines = text.split("\n") - r = f(lines, rng) - return None if r is None else "\n".join(r) - - -def comment_line(text, rng, n): - def f(lines, rng): - i = rng.randrange(len(lines) + 1) - indent = re.match(r"\s*", lines[i] if i < len(lines) else "").group(0) - return lines[:i] + [f"{indent}// fuzz {n}"] + lines[i:] - return edit_lines(text, rng, f) - - -def blank_line(text, rng, n): - return edit_lines(text, rng, lambda lines, rng: (lambda i: lines[:i] + [""] + lines[i:])(rng.randrange(len(lines) + 1))) - - -def remove_comment(text, rng, n): - def f(lines, rng): - idx = [i for i, l in enumerate(lines) if l.strip().startswith("//") and not l.strip().startswith("//!")] - if not idx: - return None - i = rng.choice(idx) - return lines[:i] + lines[i + 1:] - return edit_lines(text, rng, f) - - -def indent_line(text, rng, n): - def f(lines, rng): - idx = [i for i, l in enumerate(lines) if l.strip()] - i = rng.choice(idx) - lines[i] = " " + lines[i] - return lines - return edit_lines(text, rng, f) - - -def swap_items(text, rng, n): - b = blocks(text) - if len(b) < 3: - return None - i = rng.randrange(1, len(b) - 1) - b[i], b[i + 1] = b[i + 1], b[i] - return "\n\n".join(b) - - -def move_item_to_end(text, rng, n): - b = blocks(text) - if len(b) < 3: - return None - i = rng.randrange(1, len(b)) - item = b.pop(i) - return "\n\n".join(b + [item.rstrip("\n")]) + "\n" - - -def delete_item(text, rng, n): - b = blocks(text) - if len(b) < 3: - return None - b.pop(rng.randrange(1, len(b))) - return "\n\n".join(b) - - -def duplicate_fn(text, rng, n): - b = blocks(text) - fns = [i for i, x in enumerate(b) if re.search(r"^(pub(\([^)]*\))? )?(const )?(async )?fn \w+", x, re.M)] - if not fns: - return None - i = rng.choice(fns) - copy = re.sub(r"\bfn (\w+)", lambda m: f"fn {m.group(1)}_fuzz{n}", b[i], count=1) - b.insert(i + 1, copy) - return "\n\n".join(b) - - -ADDITIONS = [ - "fn fuzz_private_{n}() -> u32 {{ {n} }}", - "pub fn fuzz_public_{n}(x: u32) -> u32 {{ x.wrapping_mul({n}) }}", - "#[inline]\npub fn fuzz_inline_{n}(x: &T) -> (T, u32) {{ (x.clone(), {n}) }}", - "pub const FUZZ_{n}: &str = \"fuzz {n}\";", - "pub static FUZZ_STATIC_{n}: [u8; 3] = [{n} as u8, 1, 2];", - "#[derive(Debug, Clone, PartialEq)]\npub struct Fuzz{n} {{ pub items: [T; N], pub tag: &'static str }}", - "pub enum FuzzEnum{n} {{ A(u32), B {{ x: i64 }}, C }}", - "pub trait FuzzTrait{n} {{ fn go(&self) -> impl Sized; const K: u32 = {n}; }}", - "pub async fn fuzz_async_{n}() -> u32 {{ {n} }}", - "pub type FuzzAlias{n} = Vec<(T, u32)>;", - "macro_rules! fuzz_macro_{n} {{ ($e:expr) => {{ $e + {n} }}; }}", - "pub mod fuzz_mod_{n} {{ pub fn inner() -> &'static str {{ \"{n}\" }} }}", -] - - -def add_item(text, rng, n): - b = blocks(text) - i = rng.randrange(1, len(b) + 1) - b.insert(i, rng.choice(ADDITIONS).format(n=n)) - return "\n\n".join(b) - - -def int_literal(text, rng, n): - ms = [m for m in re.finditer(r"(?= args.keep: return @@ -538,6 +354,8 @@ def report(kind, detail, inc, clean, extra=None): if __name__ == "__main__": WORK.mkdir(parents=True, exist_ok=True) + for f in ("STOP", "PAUSED"): + (WORK / f).unlink(missing_ok=True) with multiprocessing.Pool(args.workers) as pool: results = pool.map(worker, range(args.workers)) total = {"edits": sum(r["edits"] for r in results), "built": sum(r["built"] for r in results), diff --git a/rustc/mutations.py b/rustc/mutations.py new file mode 100644 index 0000000..df6a2a6 --- /dev/null +++ b/rustc/mutations.py @@ -0,0 +1,201 @@ +"""Random mechanical edits to Rust source, shared by fuzz.py and flag-walk.py. + +Each edit takes the file's text, a random.Random and a counter, and returns the new text, or +None when it does not apply. EDITS lists them with weights. +""" + +import re + + + +def blocks(text): + """Top-level items: blank-line separated, continuation blocks merged.""" + out = [] + for b in text.split("\n\n"): + if out and (b[:1].isspace() or b.startswith("}") or b.startswith("where")): + out[-1] += "\n\n" + b + else: + out.append(b) + return out + + +def edit_lines(text, rng, f): + lines = text.split("\n") + r = f(lines, rng) + return None if r is None else "\n".join(r) + + +def comment_line(text, rng, n): + def f(lines, rng): + i = rng.randrange(len(lines) + 1) + indent = re.match(r"\s*", lines[i] if i < len(lines) else "").group(0) + return lines[:i] + [f"{indent}// fuzz {n}"] + lines[i:] + return edit_lines(text, rng, f) + + +def blank_line(text, rng, n): + return edit_lines(text, rng, lambda lines, rng: (lambda i: lines[:i] + [""] + lines[i:])(rng.randrange(len(lines) + 1))) + + +def remove_comment(text, rng, n): + def f(lines, rng): + idx = [i for i, l in enumerate(lines) if l.strip().startswith("//") and not l.strip().startswith("//!")] + if not idx: + return None + i = rng.choice(idx) + return lines[:i] + lines[i + 1:] + return edit_lines(text, rng, f) + + +def indent_line(text, rng, n): + def f(lines, rng): + idx = [i for i, l in enumerate(lines) if l.strip()] + i = rng.choice(idx) + lines[i] = " " + lines[i] + return lines + return edit_lines(text, rng, f) + + +def swap_items(text, rng, n): + b = blocks(text) + if len(b) < 3: + return None + i = rng.randrange(1, len(b) - 1) + b[i], b[i + 1] = b[i + 1], b[i] + return "\n\n".join(b) + + +def move_item_to_end(text, rng, n): + b = blocks(text) + if len(b) < 3: + return None + i = rng.randrange(1, len(b)) + item = b.pop(i) + return "\n\n".join(b + [item.rstrip("\n")]) + "\n" + + +def delete_item(text, rng, n): + b = blocks(text) + if len(b) < 3: + return None + b.pop(rng.randrange(1, len(b))) + return "\n\n".join(b) + + +def duplicate_fn(text, rng, n): + b = blocks(text) + fns = [i for i, x in enumerate(b) if re.search(r"^(pub(\([^)]*\))? )?(const )?(async )?fn \w+", x, re.M)] + if not fns: + return None + i = rng.choice(fns) + copy = re.sub(r"\bfn (\w+)", lambda m: f"fn {m.group(1)}_fuzz{n}", b[i], count=1) + b.insert(i + 1, copy) + return "\n\n".join(b) + + +ADDITIONS = [ + "fn fuzz_private_{n}() -> u32 {{ {n} }}", + "pub fn fuzz_public_{n}(x: u32) -> u32 {{ x.wrapping_mul({n}) }}", + "#[inline]\npub fn fuzz_inline_{n}(x: &T) -> (T, u32) {{ (x.clone(), {n}) }}", + "pub const FUZZ_{n}: &str = \"fuzz {n}\";", + "pub static FUZZ_STATIC_{n}: [u8; 3] = [{n} as u8, 1, 2];", + "#[derive(Debug, Clone, PartialEq)]\npub struct Fuzz{n} {{ pub items: [T; N], pub tag: &'static str }}", + "pub enum FuzzEnum{n} {{ A(u32), B {{ x: i64 }}, C }}", + "pub trait FuzzTrait{n} {{ fn go(&self) -> impl Sized; const K: u32 = {n}; }}", + "pub async fn fuzz_async_{n}() -> u32 {{ {n} }}", + "pub type FuzzAlias{n} = Vec<(T, u32)>;", + "macro_rules! fuzz_macro_{n} {{ ($e:expr) => {{ $e + {n} }}; }}", + "pub mod fuzz_mod_{n} {{ pub fn inner() -> &'static str {{ \"{n}\" }} }}", +] + + +def add_item(text, rng, n): + b = blocks(text) + i = rng.randrange(1, len(b) + 1) + b.insert(i, rng.choice(ADDITIONS).format(n=n)) + return "\n\n".join(b) + + +def int_literal(text, rng, n): + ms = [m for m in re.finditer(r"(?/dev/null || true; git worktree prune V="compiler/rustc_codegen_llvm/src/back/llvm_backend.rs compiler/rustc_codegen_llvm/src/base.rs compiler/rustc_codegen_llvm/src/llvm/ffi.rs compiler/rustc_codegen_ssa/src/base.rs compiler/rustc_codegen_ssa/src/traits/backend.rs compiler/rustc_incremental/src/persist/save.rs compiler/rustc_metadata/src/rmeta/encoder.rs compiler/rustc_middle/src/hooks.rs compiler/rustc_middle/src/query/on_disk_cache.rs compiler/rustc_query_impl/src/incremental.rs compiler/rustc_query_impl/src/lib.rs" @@ -31,6 +33,7 @@ git apply $H/generics-index-map.patch $H/alloc-dedup-on-decode.patch $H/metadata git add -A cd $RUST; for f in $(git diff --name-only) compiler/rustc_data_structures/src/untracked.rs; do cp $f $T/$f; done cd $T; git apply -R $H/debuginfo-checksum-stopgap.patch; git apply -R $H/threads-def-order-stopgap.patch +for p in $STOPGAPS; do git apply -R $H/$p.patch; done git add -N compiler/rustc_data_structures/src/untracked.rs git diff > $H/report-untracked.patch # 3. check the stack reproduces the tree @@ -38,6 +41,7 @@ git checkout -q HEAD -- .; git reset -q; rm -f compiler/rustc_data_structures/sr git apply $H/verify-reuse.patch git apply $H/generics-index-map.patch $H/alloc-dedup-on-decode.patch $H/metadata-source-files.patch git apply $H/report-untracked.patch; git apply $H/debuginfo-checksum-stopgap.patch $H/threads-def-order-stopgap.patch +for p in $STOPGAPS; do git apply $H/$p.patch; done n=0; cd $RUST; for f in $(git diff --name-only) compiler/rustc_data_structures/src/untracked.rs; do cmp -s $f $T/$f || { echo "differs: $f"; n=$((n+1)); }; done echo "$n files differ"; wc -l $H/verify-reuse.patch $H/report-untracked.patch | head -2 git worktree remove --force $T