Remap local def ids for deterministic metadata encoding by aerooneqq · Pull Request #162809 · rust-lang/rust

aerooneqq · 2026-09-15T12:17:30Z

Remaps local def ids that are allocated after Definitions::commit_end_of_determinism was called.
Fixes #162202, fixes #162203.

cc #t-compiler/const-eval > DefId reproducibility with the parallel frontend

r? @petrochenkov

susitsm · 2026-09-15T13:50:42Z

Are the changes in decoder.rs and def_path_hash_map.rs needed? It seems unrelated to me.

aerooneqq · 2026-09-15T14:01:33Z

Are the changes in decoder.rs and def_path_hash_map.rs needed? It seems unrelated to me.

We need to update map from hashes to def indices while encoding DefPathHashMap, now I inserted simple FxHashMap to test the approach, if I don't remove functions from decoder.rs there will be warnings, because they are not used anywhere else.

petrochenkov · 2026-09-16T12:21:55Z

@bors try @rust-timer queue

Remap def indices for deterministic metadata encoding

rust-bors · 2026-09-16T13:29:18Z

☀️ Try build successful (CI)
Build commit: ceb9839 (ceb98398abf30a57ca7f186c152d2adda7b079f9)
Base parent: 28e8a8c (28e8a8c81bf3b37909edac6c2a76e56f30cd492f)

rust-timer · 2026-09-16T14:13:41Z

Finished benchmarking commit (ceb9839): comparison URL.

Overall result: ❌ regressions - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	4.4%	[0.1%, 164.6%]	276
Regressions ❌ (secondary)	11.8%	[0.2%, 130.1%]	350
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	4.4%	[0.1%, 164.6%]	276

Max RSS (memory usage)

Results (primary 3.0%, secondary 4.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.0%	[0.8%, 9.4%]	203
Regressions ❌ (secondary)	4.6%	[0.9%, 14.8%]	327
Improvements ✅ (primary)	-2.3%	[-2.3%, -2.3%]	1
Improvements ✅ (secondary)	-3.6%	[-4.0%, -2.9%]	3
All ❌✅ (primary)	3.0%	[-2.3%, 9.4%]	204

Cycles

Results (primary 10.2%, secondary 14.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	10.2%	[1.7%, 73.0%]	55
Regressions ❌ (secondary)	14.6%	[1.4%, 66.2%]	179
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	10.2%	[1.7%, 73.0%]	55

Binary size

Results (primary -0.4%, secondary -0.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.3%]	8
Improvements ✅ (secondary)	-0.4%	[-0.4%, -0.4%]	4
All ❌✅ (primary)	-0.4%	[-0.4%, -0.3%]	8

Bootstrap: 496.137s -> 496.27s (0.03%)
Artifact size: 406.79 MiB -> 406.77 MiB (-0.01%)

petrochenkov · 2026-09-16T14:16:05Z

@bors try @rust-timer queue

Remap def indices for deterministic metadata encoding

susitsm · 2026-09-16T14:34:22Z

+
+    #[inline]
+    fn map_index(&self, index: DefIndex) -> DefIndex {
+        self.def_indexes_remapping.get(&index).copied().unwrap_or(index)


This could be donewith a vec for better performance. Since we know that these indices are dense in a range. Either a full IndexVec mapping every LocalDefId, or a shorter Vec, that only contains the remapped ones, indexed by remapped_index - min_remapped_index

View changes since the review

It is very strong assumption that def ids that are needed to be remapped will be in the end, now it should be true, because resolution and AST -> HIR lowering is executed without any parallelism, however AST -> HIR lowering became more incremental recently by introducing lower_to_hir query which is very easy to parallelize in the beginning of the run_required_analysis function (now it is executed sequentially as we prefetch hir_crate_items which is a sequential HIR visit). If it is parallelized then from each lower_to_hir execution we may invoke arbitrary code that can allocate def ids that do not need to be remapped.

But I like the idea, I was thinking about something similar in a bit different direction: create separate IndexVecs in Definitions struct for different stages of compilation, thus we can maybe pre-allocate and remap ids only for selected parts.

I think that it is reasonable to track the last def id that is allocated during resolution stage, because it is not likely to get parallelized in the nearest time (I hope so) and then to remap all def ids that are allocated after it, I will try it tomorrow.

susitsm · 2026-09-16T14:45:54Z

+    let mut to_remap = vec![];
+    let mut def_ids = vec![];
+    for idx in 0..defs.num_definitions() {
+        let def_id = LocalDefId { local_def_index: idx.into() };


I think this could iterate backwards, stopping when it hits an index not needing remapping. Remapped indices should be at the end

View changes since the review

rust-bors · 2026-09-16T15:16:50Z

☀️ Try build successful (CI)
Build commit: 8bcd37a (8bcd37a7bc50acb44dc7181f601e93127ebf19af)
Base parent: 28e8a8c (28e8a8c81bf3b37909edac6c2a76e56f30cd492f)

rust-timer · 2026-09-16T16:35:40Z

Finished benchmarking commit (8bcd37a): comparison URL.

Overall result: ❌ regressions - BENCHMARK(S) FAILED

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

❗ ❗ ❗ ❗ ❗
Warning ⚠️: The following benchmark(s) failed to build:

Job failure
cranelift-codegen-0.119.0

❗ ❗ ❗ ❗ ❗

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.1%, 0.6%]	60
Regressions ❌ (secondary)	0.2%	[0.1%, 0.3%]	13
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.1%, 0.6%]	60

Max RSS (memory usage)

Results (secondary -4.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.1%	[-5.1%, -3.3%]	3
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary -0.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	4.6%	[4.6%, 4.6%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.5%	[-2.7%, -2.3%]	2
All ❌✅ (primary)	-	-	0

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 496.137s -> 495.635s (-0.10%)
Artifact size: 406.79 MiB -> 406.74 MiB (-0.01%)

@petrochenkov

…etrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang/rust#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang/rust#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

@petrochenkov

…nkov Use `LocalDefId` as an index in metadata encoding The goal of this PR to eliminate encoding of a raw `DefIndex`, instead we should encode something (`LocalDefId`) that certainly indicates that index belongs to the local crate. The implementation of `DefIndex` encoding should always panic to prevent accidental remappings of non-local `DefIndex`es in #162809. So the plan is as follows: - Use `LocalDefId` as a writing index in tables for metadata encoding, use `LocalDefId` instead of `DefIndex` everywhere, specialize encode/decode methods to correctly process it, - Replace implementation of `encode_def_index` in `EncodeContext` with panic, so we never remap and encode def index, - Make tables' keys two-sided: we can read and write using different index types, in this case we will write using `LocalDefId` and read using `DefIndex` (done in #163447). Blocked by #162900. r? @petrochenkov

@petrochenkov

…nkov Use `LocalDefId` as an index in metadata encoding The goal of this PR to eliminate encoding of a raw `DefIndex`, instead we should encode something (`LocalDefId`) that certainly indicates that index belongs to the local crate. The implementation of `DefIndex` encoding should always panic to prevent accidental remappings of non-local `DefIndex`es in #162809. So the plan is as follows: - Use `LocalDefId` as a writing index in tables for metadata encoding, use `LocalDefId` instead of `DefIndex` everywhere, specialize encode/decode methods to correctly process it, - Replace implementation of `encode_def_index` in `EncodeContext` with panic, so we never remap and encode def index, - Make tables' keys two-sided: we can read and write using different index types, in this case we will write using `LocalDefId` and read using `DefIndex` (done in #163447). Blocked by #162900. r? @petrochenkov

aerooneqq · 2026-10-08T16:09:28Z

@rustbot ready
@bors try @rust-timer queue

Remap def indices for deterministic metadata encoding

susitsm · 2026-10-08T17:02:31Z

+        non_det_ids.push((def_id, defs.def_path_hash(def_id).local_hash()));
+    }
+
+    non_det_ids.sort_by_key(|(_, hash)| *hash);


This sort does not respect the requirements of def_id_partial_cmp.

Maybe we could save the level of the defids as the level of their parents+1? And use it in the sort

View changes since the review

Is it possible that two non-deterministic def id will be allocated in scope of a one parent and thus this sort would not be applicable here? Def path hashes are always deterministic and quite easy to use.

Never mind, I got your point.

rust-bors · 2026-10-08T17:12:17Z

☀️ Try build successful (CI)
Build commit: 8fb84f8 (8fb84f80e6ad98cae7bf15c2940bf24afc03c0ba)
Base parent: 65efa19 (65efa194201c67ba29b7384255a534d61c2aa3cb)

rust-timer · 2026-10-08T17:58:21Z

Finished benchmarking commit (8fb84f8): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up.

@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.2%	[-0.2%, -0.2%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (primary -2.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.4%	[-2.4%, -2.4%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-2.4%	[-2.4%, -2.4%]	1

Cycles

Results (primary 3.6%, secondary -1.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.6%	[3.6%, 3.6%]	1
Regressions ❌ (secondary)	3.2%	[3.2%, 3.2%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.8%	[-3.6%, -2.5%]	4
All ❌✅ (primary)	3.6%	[3.6%, 3.6%]	1

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 485.904s -> 491.874s (1.23%)
Artifact size: 406.44 MiB -> 406.64 MiB (0.05%)

susitsm · 2026-10-08T18:23:02Z

    symbol_index_table: FxHashMap<u32, usize>,
+    // Remapping of non-deterministic local def ids for stable encoding
+    // during parallel compilation.
+    local_def_ids_remapping: FxHashMap<DefIndex, DefIndex>,


We could improve perf if this is a Vec<DefIndex> where the mapping is def_index -> local_def_ids_remapping[def_index - first_non_deterministic_index] . Currently, this isn't used much, but things like #163495 can increase the amount of remapped LocalDefId-s by a lot. Could be part of the reason for the icount regressions there?

View changes since the review

rustbot assigned petrochenkov Sep 15, 2026

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 15, 2026

aerooneqq changed the title ~~Remap def indices for deterministic metadata encoing~~ Remap def indices for deterministic metadata encoding Sep 15, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Sep 16, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Sep 16, 2026

Auto merge of #162809 - aerooneqq:def-index-remapping, r=<try>

ceb9839

Remap def indices for deterministic metadata encoding

This comment has been minimized.

Sign in to view

aerooneqq force-pushed the def-index-remapping branch from cc58a69 to 9736cd8 Compare September 16, 2026 14:03

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Sep 16, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Sep 16, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Sep 16, 2026

Auto merge of #162809 - aerooneqq:def-index-remapping, r=<try>

8bcd37a

Remap def indices for deterministic metadata encoding

susitsm reviewed Sep 16, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

petrochenkov mentioned this pull request Sep 16, 2026

Parallel frontend: reproducible def ids for RPITITs #162555

Open

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Sep 16, 2026

aerooneqq force-pushed the def-index-remapping branch from 4f2feb6 to e01cdf1 Compare September 17, 2026 11:54

susitsm mentioned this pull request Oct 6, 2026

parallel compiler: generics_of encoding not reproducible #163878

Open

aerooneqq mentioned this pull request Oct 8, 2026

Parallelize AST -> HIR lowering #163495

Draft

This comment has been minimized.

Sign in to view

aerooneqq closed this Oct 8, 2026

aerooneqq force-pushed the def-index-remapping branch from 1371400 to 65efa19 Compare October 8, 2026 16:02

rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Oct 8, 2026

aerooneqq reopened this Oct 8, 2026

rustbot added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Oct 8, 2026

Remap local def ids for deterministic metadata encoding

eb934f8

aerooneqq force-pushed the def-index-remapping branch from a1c2e5a to eb934f8 Compare October 8, 2026 16:05

rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Oct 8, 2026

This comment has been minimized.

Sign in to view

aerooneqq marked this pull request as ready for review October 8, 2026 16:09

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Oct 8, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Oct 8, 2026

Auto merge of #162809 - aerooneqq:def-index-remapping, r=<try>

8fb84f8

Remap def indices for deterministic metadata encoding

aerooneqq changed the title ~~Remap def indices for deterministic metadata encoding~~ Remap local def ids for deterministic metadata encoding Oct 8, 2026

This comment has been minimized.

Sign in to view

Fix typos

5199056

susitsm reviewed Oct 8, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 8, 2026

susitsm reviewed Oct 8, 2026

View reviewed changes

Conversation

aerooneqq commented Sep 15, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

susitsm commented Sep 15, 2026

Uh oh!

aerooneqq commented Sep 15, 2026

Uh oh!

petrochenkov commented Sep 16, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Sep 16, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Sep 16, 2026

Overall result: ❌ regressions - please read:

Uh oh!

petrochenkov commented Sep 16, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

susitsm Sep 16, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

susitsm Sep 16, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rust-bors Bot commented Sep 16, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Sep 16, 2026

Overall result: ❌ regressions - BENCHMARK(S) FAILED

Uh oh!

This comment has been minimized.

aerooneqq commented Oct 8, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

susitsm Oct 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aerooneqq Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rust-bors Bot commented Oct 8, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Oct 8, 2026

Overall result: ✅ improvements - no action needed

Uh oh!

susitsm Oct 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

aerooneqq commented Sep 15, 2026 •

edited by rustbot

Loading

susitsm Sep 16, 2026 •

edited by rustbot

Loading

susitsm Sep 16, 2026 •

edited by rustbot

Loading

susitsm Oct 8, 2026 •

edited by rustbot

Loading

aerooneqq Oct 8, 2026 •

edited

Loading

susitsm Oct 8, 2026 •

edited by rustbot

Loading