Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a new internity crate providing compact string interning with both a single-threaded (Lexicon) and concurrent (ThreadedLexicon) implementation, plus a frozen Reader for fast lock-free resolution. The PR also adds extensive tests, benchmarks, and performance documentation to support correctness and compare against other Rust interners.
Changes:
- Added
internitycrate (no-std + alloc baseline, optionalstdfor concurrency, optionalserde) withSymhandles and a sealedReaderabstraction. - Added correctness coverage via integration tests, property tests (Bolero), and concurrency modeling (Loom).
- Added benchmarking + reporting tooling (Criterion, Gungraun/Callgrind, memory footprint harness) and performance/comparison docs.
Reviewed changes
Copilot reviewed 30 out of 33 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Adds internity to the repo’s primary crates list. |
| CHANGELOG.md | Links the new internity crate changelog from the workspace changelog index. |
| Cargo.toml | Adds workspace dependency entries needed by internity (and competitors used in benches). |
| Cargo.lock | Records resolved dependencies for the new crate and benchmark competitors. |
| .spelling | Adds terminology used by internity docs/code (e.g., CSR, dedup, sharded). |
| crates/internity/Cargo.toml | Defines the new crate’s features, dependencies, benches, and Loom test target. |
| crates/internity/src/lib.rs | Crate root: public API exports, feature gating, and high-level documentation. |
| crates/internity/src/lexicon.rs | Implements the single-threaded interner with dense Sym encoding + freeze to Reader. |
| crates/internity/src/threaded_lexicon.rs | Implements the concurrent sharded interner and freeze-to-reader logic. |
| crates/internity/src/sym.rs | Defines Sym handle representation and encoding/decoding helpers. |
| crates/internity/src/reader.rs | Defines the sealed Reader trait for frozen, thread-safe resolution. |
| crates/internity/src/flat_reader.rs | Frozen reader implementation for Lexicon’s dense layout. |
| crates/internity/src/shard.rs | Shard wrapper using RwLock/upgradable-read for concurrent interning. |
| crates/internity/src/shard_write.rs | Shard write-state: dedup table + CSR offsets + byte buffer. |
| crates/internity/src/shard_reader.rs | Frozen per-shard storage and resolution routine. |
| crates/internity/src/sharded_reader.rs | Frozen reader implementation for ThreadedLexicon (shard + local decode). |
| crates/internity/src/storage.rs | Centralizes unchecked UTF-8 reconstruction for fast resolve paths. |
| crates/internity/src/symbol_map.rs | Adds Sym-keyed map/set support via a specialized hasher. |
| crates/internity/src/serde_impls.rs | Implements serde for Sym and interners (feature-gated). |
| crates/internity/tests/basic.rs | Broad integration tests covering API behavior, freezing, concurrency, serde, etc. |
| crates/internity/tests/bolero_internity.rs | Property tests for dedup/resolve/freezing and raw-handle range checks. |
| crates/internity/tests/loom_internity.rs | Loom model tests for concurrent dedup and snapshot behavior. |
| crates/internity/benches/compare.rs | Criterion wall-clock benchmark suite comparing multiple interners and thread counts. |
| crates/internity/benches/counts/main.rs | Gungraun/Callgrind harness entrypoint (Linux-only runner). |
| crates/internity/benches/counts/linux.rs | Deterministic instruction/cache-count benches for hot paths + competitors. |
| crates/internity/benches/mem.rs | Memory footprint benchmark using a tracking global allocator. |
| crates/internity/scripts/perf_report.rs | Script to run benches and regenerate docs/PERF.md. |
| crates/internity/docs/PERF.md | Generated performance report output committed to the repo. |
| crates/internity/docs/COMPARISON.md | Detailed comparative analysis across the Rust string interner ecosystem. |
| crates/internity/README.md | Crate README content for docs.rs / GitHub browsing. |
| crates/internity/CHANGELOG.md | Per-crate changelog for internity. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
| Crate | Baseline | Baseline commit | This PR | Minimum required | Status |
|---|---|---|---|---|---|
internity |
new crate | — | 0.1.0 | 0.1.0 | ✅ ok |
internity_macros |
new crate | — | 0.1.0 | — | |
internity_macros_impl |
new crate | — | 0.1.0 | 0.1.0 | ✅ ok |
⚠️ internity_macros — manual proc-macro review detail
`internity_macros` is a proc-macro-only crate. cargo-semver-checks intentionally skips proc-macro targets because they have no supported library API surface. Review exported macro names, accepted input syntax, diagnostics, and generated output manually; build and test results do not establish public API SemVer compatibility.
Proc-macro API compatibility must be reviewed manually; successful builds and tests do not establish SemVer compatibility. This check is informational and does not block the merge.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #604 +/- ##
=========================================
Coverage 100.0% 100.0%
=========================================
Files 444 470 +26
Lines 43456 45302 +1846
=========================================
+ Hits 43456 45302 +1846
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 64 out of 65 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (6)
crates/internity/src/local_lexicon.rs:317
- The crate uses a coverage gate that expects
#[cfg_attr(coverage_nightly, coverage(off))]on each#[cfg(test)] mod testsso test-only lines don’t count toward coverage. This module’s tests are missing that attribute and may fail the workspace coverage check.
#[cfg(test)]
mod tests {
crates/internity/src/threaded_lexicon.rs:318
- The crate uses a coverage gate that expects
#[cfg_attr(coverage_nightly, coverage(off))]on each#[cfg(test)] mod testsso test-only lines don’t count toward coverage. This module’s tests are missing that attribute and may fail the workspace coverage check.
crates/internity/src/symbol_map.rs:72 - The crate uses a coverage gate that expects
#[cfg_attr(coverage_nightly, coverage(off))]on each#[cfg(test)] mod testsso test-only lines don’t count toward coverage. This module’s tests are missing that attribute and may fail the workspace coverage check.
crates/internity/src/storage.rs:62 - The crate uses a coverage gate that expects
#[cfg_attr(coverage_nightly, coverage(off))]on each#[cfg(test)] mod testsso test-only lines don’t count toward coverage. This module’s tests are missing that attribute and may fail the workspace coverage check.
crates/internity/src/sym.rs:160 - The crate uses a coverage gate that expects
#[cfg_attr(coverage_nightly, coverage(off))]on each test module so test-only lines don’t count toward coverage. This test module is missing that attribute and may fail the workspace coverage check.
.spelling:755 .spellingaddsQuiesce, but the codebase uses lowercasequiescein documentation. The Hunspell dictionary is case-sensitive in practice, so add the lowercase form as well to avoid spellcheck failures.
Quiesce
transactional
snapshotted
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 64 out of 65 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (6)
crates/internity/src/local_lexicon.rs:317
#[cfg(test)] mod testsblocks count toward the workspace’s 100% coverage gate unless they’re excluded undercoverage_nightly. Other crates consistently add#[cfg_attr(coverage_nightly, coverage(off))]above test modules (e.g. crates/tick/src/error.rs:103-105). Add the attribute here to avoid coverage-gate failures.
#[cfg(test)]
mod tests {
crates/internity/src/threaded_lexicon.rs:318
#[cfg(test)] mod testsblocks count toward the workspace’s 100% coverage gate unless they’re excluded undercoverage_nightly. Other crates consistently add#[cfg_attr(coverage_nightly, coverage(off))]above test modules (e.g. crates/tick/src/error.rs:103-105). Add the attribute here to avoid coverage-gate failures.
crates/internity/src/sym.rs:160- This
testsmodule is gated on#[cfg(all(test, feature = "std"))]but will still be counted by the workspace’s 100% coverage gate when it’s compiled. Other crates exclude test modules undercoverage_nightly(e.g. crates/tick/src/error.rs:103-105). Add#[cfg_attr(coverage_nightly, coverage(off))]above this module to prevent coverage-gate failures.
crates/internity/src/storage.rs:62 #[cfg(test)] mod testsblocks count toward the workspace’s 100% coverage gate unless they’re excluded undercoverage_nightly. Other crates consistently add#[cfg_attr(coverage_nightly, coverage(off))]above test modules (e.g. crates/tick/src/error.rs:103-105). Add the attribute here to avoid coverage-gate failures.
crates/internity/src/symbol_map.rs:72#[cfg(test)] mod testsblocks count toward the workspace’s 100% coverage gate unless they’re excluded undercoverage_nightly. Other crates consistently add#[cfg_attr(coverage_nightly, coverage(off))]above test modules (e.g. crates/tick/src/error.rs:103-105). Add the attribute here to avoid coverage-gate failures.
crates/internity/src/de/impls.rs:453#[cfg(test)] mod testsblocks count toward the workspace’s 100% coverage gate unless they’re excluded undercoverage_nightly. Other crates consistently add#[cfg_attr(coverage_nightly, coverage(off))]above test modules (e.g. crates/tick/src/error.rs:103-105). Add the attribute here to avoid coverage-gate failures.
#[cfg(test)]
mod tests {
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 64 out of 65 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (6)
crates/internity/src/sym.rs:160
- Unit-test modules should be excluded from the coverage gate by adding
#[cfg_attr(coverage_nightly, coverage(off))]above themod testsblock; otherwise test-only lines can count against the per-crate 100% coverage requirement.
crates/internity/src/threaded_lexicon.rs:370 - Unit-test modules should be excluded from the coverage gate by adding
#[cfg_attr(coverage_nightly, coverage(off))]above themod testsblock; otherwise test-only lines can count against the per-crate 100% coverage requirement.
crates/internity/src/symbol_map.rs:72 - Unit-test modules should be excluded from the coverage gate by adding
#[cfg_attr(coverage_nightly, coverage(off))]above themod testsblock; otherwise test-only lines can count against the per-crate 100% coverage requirement.
crates/internity/src/shard.rs:28 - The intra-doc link points at the private module path
crate::threaded_lexicon::..., which is brittle and can break documentation links. Prefer the public re-export pathcrate::ThreadedLexicon::freeze.
crates/internity/src/local_lexicon.rs:378 - Unit-test modules should be excluded from the coverage gate by adding
#[cfg_attr(coverage_nightly, coverage(off))]above themod testsblock; otherwise test-only lines can count against the per-crate 100% coverage requirement.
#[cfg(test)]
mod tests {
crates/internity/src/storage.rs:62
- Unit-test modules should be excluded from the coverage gate by adding
#[cfg_attr(coverage_nightly, coverage(off))]above themod testsblock; otherwise test-only lines can count against the per-crate 100% coverage requirement.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 64 out of 65 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (5)
crates/internity/src/threaded_lexicon.rs:371
- Repository coverage gating expects internal
#[cfg(test)] mod testsblocks to be excluded from coverage with#[cfg_attr(coverage_nightly, coverage(off))]. Without it, these unit-test-only lines count against the 100% per-package gate and can fail CI even when production code is fully covered.
crates/internity/src/sym.rs:160 - Unit-test-only modules should be excluded from the per-package line coverage gate via
#[cfg_attr(coverage_nightly, coverage(off))]. Otherwise these lines can show up as uncovered when tests don’t exercise every branch inside the test module itself.
crates/internity/src/symbol_map.rs:72 - Add
#[cfg_attr(coverage_nightly, coverage(off))]above this#[cfg(test)] mod testsblock so these test-only lines don’t count against the repository’s 100% coverage gate undercoverage_nightly.
crates/internity/src/storage.rs:62 - This crate enables
coverage_nightly; to keep the workspace 100% line gate stable, unit-test-only modules should be annotated with#[cfg_attr(coverage_nightly, coverage(off))]so their lines don’t get counted as uncovered.
crates/internity/src/local_lexicon.rs:391 - Please mark this
#[cfg(test)] mod testswith#[cfg_attr(coverage_nightly, coverage(off))]to avoid test-module lines counting toward the 100% coverage gate.
#[cfg(test)]
mod tests {
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 73 out of 74 changed files in this pull request and generated 8 comments.
Comments suppressed due to low confidence (1)
crates/internity_macros_impl/src/lib.rs:56
- The workspace coverage gate counts lines inside
#[cfg(test)]modules unless they’re excluded undercoverage_nightly. Add#[cfg_attr(coverage_nightly, coverage(off))]above this test-only module so it doesn’t affect the per-crate 100% line gate (see e.g. crates/tick/src/error.rs:103-105).
Add compact local and concurrent string interners with frozen, lock-free
readers, no-std and Serde support, property tests, Loom models, and Criterion,
Gungraun, and memory benchmarks.
Add `intern_bytes(&[u8]) -> Result<Sym, Utf8Error>` on both engines (and the
`Lexicon` trait): it accepts raw bytes and validates UTF-8 only on a dedup miss,
skipping the check when the input byte-matches an already-interned (hence valid)
string. This amortizes UTF-8 validation across duplicate inserts — each distinct
string is validated once — while handles still resolve to a checked `&str`.
Add an interner-aware, symmetric Serde model: `SerializeIn` resolves every `Sym`
to its string against a frozen `Reader` (round-trippable, no raw handles on the
wire), and `DeserializeIn` interns string fields straight into a chosen
`LocalLexicon` or `ThreadedLexicon`. Both come with `#[derive(SerializeIn)]` /
`#[derive(DeserializeIn)]` and cover scalars, `Option`, `Box`, `Vec`, tuples, the
`BTree{Map,Set}` and (under `std`) `Hash{Map,Set}` containers, including the
`SymMap` / `SymSet` aliases. `SerializeIn` is `?Sized` in its reader, so an erased
`&dyn Reader` works; serde's `skip_serializing_if` is a compile-time error on the
serialize side. `SerializeReader` snapshots a whole frozen corpus.
Benchmarks below run single-threaded over a corpus of ~6000 identifier-like
strings; lower is better. See `crates/internity/docs/PERF.md` for the full
matrix (multi-threaded 1/2/4/8-thread flavors plus Callgrind instruction counts)
and `docs/COMPARISON.md` for the ecosystem analysis. In short: internity leads
`lasso` on concurrent insert at every measured thread count and stays far ahead
of `symbol_table` on concurrent lookup, while `lasso`'s frozen resolver edges
ahead on concurrent lookup at 1-4 threads before internity retakes the lead at 8.
insert — cold interning (single-threaded):
| Interner | Time | Δ vs internity |
|---|---:|---:|
| internity | 259.40 µs | ref |
| internity (threaded) | 459.03 µs | +77.0% |
| lasso | 623.22 µs | +140.3% |
| string-interner | 323.61 µs | +24.8% |
| symbol_table | 504.68 µs | +94.6% |
reuse — repeated dedupe hits (single-threaded):
| Interner | Time | Δ vs internity |
|---|---:|---:|
| internity | 110.33 µs | ref |
| internity (threaded) | 181.17 µs | +64.2% |
| lasso | 287.61 µs | +160.7% |
| string-interner | 136.56 µs | +23.8% |
| symbol_table | 174.09 µs | +57.8% |
| ustr | 312.15 µs | +182.9% |
| string_cache | 279.44 µs | +153.3% |
lookup — resolve handle to `&str` (single-threaded):
| Interner | Time | Δ vs internity |
|---|---:|---:|
| internity (live) | 12.00 µs | ref |
| internity (frozen) | 11.44 µs | -4.7% |
| lasso | 9.92 µs | -17.4% |
| string-interner | 14.53 µs | +21.1% |
| symbol_table | 49.70 µs | +314.1% |
| ustr | 7.49 µs | -37.6% |
| string_cache | 10.22 µs | -14.8% |
Live heap held (owned interners; filled interner vs frozen read form):
| Interner | Handle | Filled | Lookup |
|---|---:|---:|---:|
| internity LocalLexicon | 4 B | 172.0 KiB | 96.5 KiB |
| internity ThreadedLexicon | 4 B | 181.1 KiB | 98.8 KiB |
| lasso | 4 B | 264.2 KiB | 224.2 KiB |
| string-interner | 4 B | 204.0 KiB | 204.0 KiB |
| symbol_table | 4 B | 241.3 KiB | 241.3 KiB |
| string_cache | 8 B | 351.7 KiB | 351.7 KiB |
Instruction counts (gungraun/Callgrind) — standalone deterministic microprobes,
NOT counterparts to the wall-clock tables above. Each measures a single
operation against a table of 1000 filler entries plus the fixed key (1001
preloaded); occupancy, key distribution, hashers, and operation granularity all
differ from the ≈6000-operation corpus loop, so these must not be read as a
per-row explanation of the timings. Mem accesses = L1 + LL + RAM hits.
insert — single operation:
| Interner | Instructions | Mem accesses |
|---|---:|---:|
| internity | 220 | 307 |
| internity (threaded) | 815 | 1,119 |
| lasso | 272 | 404 |
| string-interner | 198 | 277 |
| symbol_table | 490 | 723 |
| ustr | 164 | 230 |
| string_cache | 654 | 821 |
reuse — single operation:
| Interner | Instructions | Mem accesses |
|---|---:|---:|
| internity | 150 | 209 |
| internity (threaded) | 162 | 224 |
| lasso | 172 | 251 |
| string-interner | 178 | 252 |
| symbol_table | 441 | 656 |
| ustr | 130 | 181 |
| string_cache | 292 | 356 |
lookup — single operation:
| Interner | Instructions | Mem accesses |
|---|---:|---:|
| internity (live) | 30 | 51 |
| internity (frozen) | 26 | 43 |
| lasso | 31 | 53 |
| string-interner | 40 | 62 |
| symbol_table | 510 | 806 |
| ustr | 4 | 7 |
| string_cache | 20 | 33 |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9e86d1af-9907-4653-a588-86d0e9eda4cd
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 73 out of 74 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (5)
crates/internity/src/storage.rs:62
- Tests modules in this repo are typically annotated with
#[cfg_attr(coverage_nightly, coverage(off))]so test-only lines don’t count against the coverage gate (e.g.crates/fetch_tls/src/native_tls.rs:90-92). This module is missing that attribute, which can break the 100% coverage CI gate.
crates/internity/src/symbol_map.rs:72 - Tests modules in this repo are typically annotated with
#[cfg_attr(coverage_nightly, coverage(off))]so test-only lines don’t count against the coverage gate (e.g.crates/fetch_tls/src/native_tls.rs:90-92). This module is missing that attribute, which can break the 100% coverage CI gate.
crates/internity/src/local_lexicon.rs:391 - Tests modules in this repo are typically annotated with
#[cfg_attr(coverage_nightly, coverage(off))]so test-only lines don’t count against the coverage gate (e.g.crates/fetch_tls/src/native_tls.rs:90-92). This module is missing that attribute, which can break the 100% coverage CI gate.
#[cfg_attr(coverage_nightly, coverage(off))]
#[cfg(test)]
crates/internity/src/sym.rs:166
- Tests modules in this repo are typically annotated with
#[cfg_attr(coverage_nightly, coverage(off))]so test-only lines don’t count against the coverage gate (e.g.crates/fetch_tls/src/native_tls.rs:90-92). This module is missing that attribute, which can break the 100% coverage CI gate.
crates/internity/src/symbol_map.rs:9 - The docs claim
HashMap/HashSetkeyed bySymhave “collision-free behavior”. Even with an injective hasher output foru32, hash tables can still have bucket collisions due to table size/modulo, so “collision-free” is inaccurate and could mislead readers.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 73 out of 74 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
crates/internity/src/se/mod.rs:37
- The example claims “the same handle comes back”, but
DeserializeIngenerally re-interns strings into a fresh lexicon and may assign different numericSymvalues (especially for multi-field values / containers, andHashMap/HashSetwhere traversal order is nondeterministic). This line should describe string round-tripping rather than handle identity.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 26e8d676-1468-4976-949a-5dca30587a30
martin-kolinek
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(internity): introduce string interning crate
Add compact local and concurrent string interners with frozen, lock-free
readers, no-std and Serde support, property tests, Loom models, and Criterion,
Gungraun, and memory benchmarks.
Add
intern_bytes(&[u8]) -> Result<Sym, Utf8Error>on both engines (and theLexicontrait): it accepts raw bytes and validates UTF-8 only on a dedup miss,skipping the check when the input byte-matches an already-interned (hence valid)
string. This amortizes UTF-8 validation across duplicate inserts — each distinct
string is validated once — while handles still resolve to a checked
&str.Add an interner-aware, symmetric Serde model:
SerializeInresolves everySymto its string against a frozen
Reader(round-trippable, no raw handles on thewire), and
DeserializeIninterns string fields straight into a chosenLocalLexiconorThreadedLexicon. Both come with#[derive(SerializeIn)]/#[derive(DeserializeIn)]and cover scalars,Option,Box,Vec, tuples, theBTree{Map,Set}and (understd)Hash{Map,Set}containers, including theSymMap/SymSetaliases.SerializeInis?Sizedin its reader, so an erased&dyn Readerworks; serde'sskip_serializing_ifis a compile-time error on theserialize side.
SerializeReadersnapshots a whole frozen corpus.Benchmarks below run single-threaded over a corpus of ~6000 identifier-like
strings; lower is better. See
crates/internity/docs/PERF.mdfor the fullmatrix (multi-threaded 1/2/4/8-thread flavors plus Callgrind instruction counts)
and
docs/COMPARISON.mdfor the ecosystem analysis. In short: internity leadslassoon concurrent insert at every measured thread count and stays far aheadof
symbol_tableon concurrent lookup, whilelasso's frozen resolver edgesahead on concurrent lookup at 1-4 threads before internity retakes the lead at 8.
insert — cold interning (single-threaded):
reuse — repeated dedupe hits (single-threaded):
lookup — resolve handle to
&str(single-threaded):Live heap held (owned interners; filled interner vs frozen read form):
Instruction counts (gungraun/Callgrind) — standalone deterministic microprobes,
NOT counterparts to the wall-clock tables above. Each measures a single
operation against a table of 1000 filler entries plus the fixed key (1001
preloaded); occupancy, key distribution, hashers, and operation granularity all
differ from the ≈6000-operation corpus loop, so these must not be read as a
per-row explanation of the timings. Mem accesses = L1 + LL + RAM hits.
insert — single operation:
reuse — single operation:
lookup — single operation: