[Zappandy]

Related: zarr-developers/zarr-python#4531.

Problem description

zarr-python 3 sets numcodecs.blosc.use_threads = False on import, so blosc.decompress calls blosc_decompress_ctx(..., 1) and one large chunk is decoded on one thread (zarr-developers/zarr-python#4531). Zarr maintainers suggested an integer nthreads argument passed to the _ctx functions.

I'm passing nthreads: compress(source, cname, clevel, ..., nthreads=None) and decompress(source, dest=None, nthreads=None). With None the number of threads is set as before, by use_threads and set_nthreads. An integer from 1 to MAX_THREADS makes the call use blosc_compress_ctx or blosc_decompress_ctx with that many threads, without _MUTEX or the global context. Other values raise ValueError.

I put a small script that times both functions for different block counts and thread counts in a gist: https://gist.github.com/Zappandy/6975e3231a64068ad541e47ea1331b5f

About compress

I've a separate commit for this one in case you guys prefer i roll it back. With nthreads greater than 1, compressing a buffer of several blocks can give different bytes between calls. THat being said, I compressed the same buffer 300 times: each block had the same bytes in every run, only the order of the blocks in the buffer changed, and the decoded data was the same. nthreads=1 gave the same bytes every time. I saw the same with use_threads=True and set_nthreads(8), and the default (use_threads=None) also uses Blosc threads when called from the main thread (_get_use_threads in blosc.pyx).

Version and installation information

• numcodecs.__version__: 0.17.1 (main), c-blosc 1.21.7.dev

• Python 3.12.13

• Linux; built from source with meson-python
  TODO:

• Unit tests and/or doctests in docstrings

• Tests pass locally

• Docstrings and API docs for any new/modified user-facing classes and functions

• Changes documented in docs/release.rst

• Docs build locally

• GitHub Actions CI passes

• Test coverage to 100% (Codecov passes)

Codevoc doesn't measure .pyx files

[d-v-b]

this is cool, and I don't think we are worried about the flaky byte representation. We just need to document what's constant, and what's not.

[Copilot]

🟢 Approval recommended

No blocking issues remain; the retained feedback is a minor clarification of fork-related documentation.

1 open issue

• Document forked-child exception for automatic thread selection · New

What changed in this PR

This PR adds per-call thread control to Blosc compression and decompression, allowing callers to request multiple threads even when global threading is disabled, while preserving existing defaults.

Changes:

• Add validated optional nthreads arguments that select Blosc’s context functions.
• Add coverage for round trips, destination buffers, invalid values, global-state isolation, concurrency, and forked workers.
• Document the new arguments and announce them in the release notes.

File	Description
tests/​test_blosc.py	Tests explicit thread counts and compatibility with existing behavior.
src/​numcodecs/​blosc.pyx	Adds argument validation, context dispatch, and API documentation.
docs/​release.md	Announces optional per-call thread control.

_{Review effort: Balanced}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (54fc274) to head (d2b912c).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #882   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines          914       914           
=========================================
  Hits           914       914           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.