compression: cap the default zstd MT workers at 4 for chunks

[ThomasWaldmann]

The BORG_ZSTD_MT_WORKERS default was the cpu count. But a chunk only yields ceil(size / 512KiB) compression jobs - 4 for the 2MiB chunks the default chunker aims at - so most of a big thread pool gets no work, while the whole pool is still created and torn down again for every single chunk.

The per-chunk default is now min(cpu count, 4).

Chunk-level worker sweep on a 12-core machine (M3 Pro), default fastcdc chunker (512KiB..8MiB chunks), real corpora, big chunks only (>= 768KiB; smaller ones are always compressed single-threaded). Throughput in MB/s at the default zstd,-4:

workers	code	binaries	entropy	VM image
1	1192	1353	5072	6510
2	1696	1985	5527	5772
4	2118	2480	4298	5217
6	2061	2388	4009	4567
8	1997	2316	3774	4267
12 (old default)	1877	2158	3383	3813

4 workers beat 12 on every corpus at zstd,-4 (+13% .. +37%). At zstd,3 the sweep peaks at 4-6 workers with 12 always below the peak, at zstd,10 it is flat from 4 workers up - no clear win for 12 anywhere. Bucketing the timings by jobs-per-chunk shows the peak tracking the job count: 2-job chunks (768KiB..1MiB) peak at 2 workers, the dominant 4-job (~2MiB) chunks at 4, so beyond the job count extra threads only add startup overhead.

export-tar is different: it compresses one long stream through a single ZstdFile, using libzstd's default (large) job size, so there are always enough jobs and the pool is created only once. It keeps the cpu count as its default via the new stream=True parameter. An explicitly set BORG_ZSTD_MT_WORKERS still overrides both defaults, e.g. for chunker configurations producing much bigger chunks.

The sweep also showed that on data zstd is very fast on anyway, single-threaded compression can beat any multi-threaded configuration at fast levels (at zstd,-4: VM image 6510 MB/s single-threaded vs 5217 at 4 workers, pure entropy 5072 vs 4298) - while typical file-tree data gains far more from multi-threading than such data loses (code +78%, binaries +83% for 4 workers vs 1). The default stays multi-threaded; the borg help environment text now mentions setting BORG_ZSTD_MT_WORKERS=1 as a speed option for such data, in addition to its existing ratio/cpu-sharing advice.

Follow-up to #10100.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.02%. Comparing base (daa1944) to head (527fc57).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10115      +/-   ##
==========================================
- Coverage   87.03%   87.02%   -0.02%     
==========================================
  Files         101      101              
  Lines       17730    17730              
  Branches     2678     2678              
==========================================
- Hits        15431    15429       -2     
- Misses       1598     1599       +1     
- Partials      701      702       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.