Skip to content

Fix slow decoding of large MIME inputs on Linux; add opt-in exact sizing; benchmark MIME (CRLF) - #5

Merged
lemire merged 4 commits into
mainfrom
mime-bench
Nov 18, 2026
Merged

lemire merged 4 commits into
mainfrom
mime-bench

Conversation

@lemire

@lemire lemire commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Problem

On Linux (glibc), decoding large MIME input (76-character lines ending with CRLF) was about four times slower than it should be: 3.1 GB/s for 1 MiB on a Xeon 6548N, against 19.7 GB/s for the same data without line breaks.

try_decode sizes the output with maximal_binary_length_from_base64, which counts line breaks as data, so MIME input always decodes to fewer bytes and the result is shrunk with _PyBytes_Resize. With glibc:

  1. a block of this size is mmap'ed, and the shrink is an mremap;
  2. freeing the result sets glibc's adaptive mmap threshold to the shrunk size;
  3. the next call requests the larger size again, above the threshold, so it maps fresh pages and faults on each of them.

strace shows one mmap, one mremap and one munmap per call, about 250 page faults per call for 1 MiB. Plain input never shrinks and is not affected; macOS malloc is not affected either.

Fix

When the slack is at most 1/8 of the buffer, shorten the bytes object in place (Py_SET_SIZE and a NUL terminator) instead of reallocating it. Every call then requests the same block size, the threshold settles on it, and the block is reused. The memory overhead is at most 1/8, about 2.6% for 76-character CRLF lines. Larger slack still goes through _PyBytes_Resize.

Exact sizing (opt-in)

fastpybase64.set_exact_sizing(True) (with get_exact_sizing()) sizes decoded outputs exactly: simdutf::binary_length_from_base64 computes the length, and base64_to_binary_safe decodes into the buffer. No memory goes unused, but the extra pass makes decoding slower, so it is off by default. The setting is process-wide (a std::atomic<bool>, safe on free-threaded builds). The whole test suite now runs in both modes, and bench.py gains --exact-sizing. The README has a new Memory section.

Xeon 6548N, GB/s of decoded data:

default exact sizing
b64decode (plain), 1 MiB 19.08 13.97
b64decode (plain), 16 MiB 14.35 9.17
b64decode(MIME CRLF), 1 MiB 12.03 8.45
b64decode(MIME CRLF), 16 MiB 12.04 7.69

Benchmark

bench.py gains a b64decode(MIME CRLF) operation. Measuring everything in one process hid the problem: earlier allocations (building the input, random.randbytes, pybase64's temporary copy) raise glibc's threshold. Each row now runs in a fresh child process that reads its input from a file, with fastpybase64 timed first and the results checked afterwards. --same-process keeps the old behavior.

Results

Xeon Gold 6548N (Emerald Rapids), one core, Python 3.15.2, icelake kernel, GB/s of decoded data:

main this PR
b64decode(MIME CRLF), 64 KiB 12.32 13.39
b64decode(MIME CRLF), 1 MiB 3.07 12.07
b64decode(MIME CRLF), 16 MiB 6.76 12.06
decodebytes, 1 MiB 3.09 9.8–12.1
b64decode (plain), 1 MiB 19.67 19.61

The standard library suffers from the same trap: its a2b_base64 also over-allocates and shrinks.

Tests (642, both modes) pass on Linux x64 and on macOS arm64 (Apple M4 Max). On the Mac, the numbers are unchanged within noise.

Measuring in one process let earlier allocations raise glibc's mmap
threshold, which hid the cost of allocating large outputs. Each row now
runs in a child process that reads its input from a file, with
fastpybase64 timed first; --same-process restores the old behavior.
MIME input decodes to fewer bytes than the maximal length, and shrinking
the result with _PyBytes_Resize trapped glibc into mmap'ing fresh pages
on every call. Decoding 1 MiB of CRLF MIME on a Xeon 6548N goes from
3.1 to 12.1 GB/s.
The default keeps the in-place shrink, which may leave up to 1/8 of the
allocation unused. Exact sizing computes the length with
binary_length_from_base64 and decodes with base64_to_binary_safe; it wastes
no memory but decodes more slowly. The tests run in both modes, and
bench.py gains --exact-sizing.
@lemire lemire changed the title Fix slow decoding of large MIME inputs on Linux; benchmark MIME (CRLF) Fix slow decoding of large MIME inputs on Linux; add opt-in exact sizing; benchmark MIME (CRLF) Nov 18, 2026
@lemire
lemire merged commit ca35185 into main Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant