[Backport 1.2-maint] crc32_clmul: only feed full aligned blocks to the folding code - #10154
Merged
Merged
Conversation
The folding implementation loads full 16 byte vectors from 16 byte aligned addresses, but it was also used for the bytes in front of the first alignment boundary and for inputs shorter than 16 bytes. That caused two defects: - For inputs of 4..15 bytes it loaded 16 bytes starting at src, reading up to 12 bytes past the end of the buffer. Where those bytes fall into an unmapped page, this segfaults. repository.py _read() calls crc32(memoryview(header)[4:]), which is a 5 byte view, so this was on the hot path for every segment entry read and is the cause of the intermittent NetBSD test suite segfaults tracked in #5922. fixes #10149 - In the alignment prologue XOR_INITIAL() xored the 32 bit initial value into the low 4 bytes of the vector, but partial_fold() then kept only algn_diff bytes of it. For algn_diff of 1..3 the upper bytes of the initial value were silently discarded, giving a wrong checksum. fixes #10150 Hand both cases to the table driven implementation instead, so the folding code only ever sees full aligned blocks and the initial value is always folded into a complete 16 byte vector. This is the structure zlib-ng uses as well, see crc32_copy_small() in their arch/x86/crc32_pclmulqdq_tpl.h. It is also faster for the sizes borg actually uses. Measured on an amd64 Zen 3 machine, ns per call: len before after 5 9.45 5.00 9 9.60 7.90 16 7.42 6.70 41 11.68 9.28 4096 248.88 246.30 Only 12..15 byte inputs get slower (9.6 -> 13.8 ns), and borg passes 5 and 37 byte headers. Tests: test_crc32 only swept start offsets 0..3. CPython's bytes payloads are 16 byte aligned, so those cover alignments 0, 15, 14 and 13 and can never reach the broken 1..3 - widen the sweep to 0..15. Add three more tests, all comparing against zlib.crc32: - test_crc32_random_slices takes memoryview slices of a 1 MiB buffer at random, unaligned start and end offsets, which gives large slices with an unaligned head at every 16 byte alignment. - test_crc32_short_slices walks every end offset from start to start+50 for 10 random starts, covering the empty input, the scalar path, the short path and the folding path at arbitrary alignments. - test_crc32_clmul_no_overread puts the buffer flush against a PROT_NONE guard page. It runs in a subprocess, so a regression fails the test rather than killing the test runner with a SIGSEGV. All four fail against the unpatched implementation and pass with this change; they add about 0.4s to the test suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 38b6dce)
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #10151 to
1.2-maint.