compression: use zstd,-4 as the new default

[ThomasWaldmann]

Makes zstd,-4 the default compression for create, recreate, import-tar, transfer, repo-compress and debug put-obj (previously lz4). Fixes #10085.

Why zstd,-4

Benchmarked on ~45 GB of real data on an M3 Pro (12 cores), with the default chunker (fastcdc, ~2 MiB target) and the current default of 4 zstd MT workers per chunk. Measured both at the compressor level (chunks fed to the compressor exactly as borg produces them) and end-to-end via borg create / borg extract. Corpora:

• code: four dev trees incl. their .git (7.6 GB, 250k chunks, 44% of bytes in small chunks < 768 KiB, i.e. no MT)
• binaries: a homebrew installation (11.0 GB, 219k chunks, 26% small-chunk bytes)
• vm: a Windows 11 qcow2 disk image (26.7 GB logical, ~17 GB allocated)
• entropy: synthetic 4-bit-per-byte data - no repeats, compressible only by entropy coding (5.4 GB)

Compressor level, byte-weighted (size = % of lz4's compressed size, lower is better):

corpus	spec	speed vs lz4	size vs lz4
code	zstd,-4	1.26x	100.2%
	zstd,-2	1.19x	93.7%
	zstd,-1	1.13x	91.0%
binaries	zstd,-4	1.41x	98.6%
	zstd,-2	1.30x	92.6%
	zstd,-1	1.27x	90.1%
entropy	zstd,-4	>= 1.2x	100%
vm	zstd,-4	0.72x	99.6%

On small chunks alone (single-threaded) lz4 keeps a slight edge; big chunks flip it, as libzstd's threads engage from 768 KiB up. On the VM image lz4 stays ahead at every negative level: that image is dominated by long, easy repeats which lz4 races through at ~8 GB/s, and no zstd level changes this (they all land within 0.70x..0.74x) because the level only steers match search, which is not the limit there.

End-to-end, median of 3 interleaved iterations, fresh repo per run, -e none-blake3:

corpus	spec	create	extract	repo size
code, 7.7 GB	lz4	57.3s	86.2s	3348 MB
	zstd,-4	56.3s	91.2s	3365 MB
	zstd,-2	56.3s	87.4s	3177 MB
	zstd,3	59.4s	85.7s	2718 MB
binaries, 19.0 GB	lz4	87.3s	93.3s	9539 MB
	zstd,-4	83.5s	96.6s	9402 MB
	zstd,-2	83.7s	101.4s	8827 MB
	zstd,3	92.7s	101.2s	7542 MB
vm, 26.7 GB	lz4	37.2s	27.0s	17263 MB
	zstd,-4	41.2s	25.5s	17196 MB
	zstd,3	43.8s	26.1s	17190 MB
entropy, 5.4 GB	lz4	9.3s	5.8s	5370 MB
	zstd,-4	9.5s	5.8s	5370 MB
	zstd,3	14.7s	9.0s	2785 MB

create with zstd,-4 is faster than lz4 on both file-tree corpora (2% on code, 5% on binaries), a tie on incompressible data, and 11% slower on the VM image - at the same or smaller repo size everywhere. extract is 5-6% slower on the tree corpora (lz4 decompresses faster in isolation, though restore is not decompression-bound) and 6% faster on the VM image.

That matches the goal in #10085: as fast as or faster than lz4 in the common cases, MT for large chunks, good small-chunk speed, and slightly better compression thrown in. Users who back up mostly hard-to-compress images can keep lz4's edge there with -C lz4, -C auto,zstd,-4 or -C none.

Why this level: levels below -4 buy no end-to-end speed at all (create is flat from -10 to -1, within ~2%) and only cost repo size - zstd,-10 stores 12% more than lz4 on code. Levels above -4 shrink repos further at unchanged create time, and positive levels cost real time (zstd,3: +4% create on code, +6% on binaries, +58% on entropy). zstd,-4 is the speed-first choice, as suspected in #10085; zstd,-2 is the ratio-leaning alternative at the same create time (repos 5-6% smaller) if that trade is preferred.

The entropy corpus also shows what the negative levels give up: they skip literal entropy coding, so data that only an entropy coder can shrink stays uncompressed for them (as for lz4), while zstd,3 halves it.

Verified functionally: after borg create without -C, a borg repo-compress -C zstd,-4 --stats run reports all compressible objects as "already had the desired compression".

Changes

• all six -C argparse defaults: lz4 -> zstd,-4
• borg help compression: default moved from the lz4 entry to zstd
• docs: quickstart, FAQ, create examples updated; usage/man pages regenerated
• borg benchmark cpu: zstd,-4 added to the compressor list
• lz4 itself is unchanged and stays available; nothing about the repo format changes (per-chunk ctype/clevel as before)

Tests

No test asserted the default spec; the one stale comment mentioning it was updated. CI is green; locally 2505 tests pass and the failures that remain reproduce identically on unmodified master in the same environment.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.04%. Comparing base (9ef3695) to head (b9da5fd).
⚠️ Report is 29 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #10100   +/-   ##
=======================================
  Coverage   87.03%   87.04%           
=======================================
  Files         101      101           
  Lines       17730    17742   +12     
  Branches     2678     2680    +2     
=======================================
+ Hits        15432    15444   +12     
+ Misses       1598     1597    -1     
- Partials      700      701    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.