Block-parallel zlib/gzip with pthreads and a block index - #12
Merged
Merged
Conversation
An earlier attempt at this lives on the pzlib branch and ends with the commit
"[deadend] multi-threaded zlib/gzip offers no speedup for memory-based buffer".
That conclusion was an artifact of two bugs, not a property of the problem:
- the block count was capped at the thread count while the block size stayed
fixed at 128 KB, so the final block received the remainder -- 88% to 98% of
the input. Maximum achievable speedup was 1.02x to 1.13x by construction,
whatever the thread count.
- nothing emitted Z_FULL_FLUSH and no offsets were recorded, so the blocks
were not independently inflatable and the decompressor was attempting the
genuinely impossible thing: inflating a slice of an ordinary deflate stream.
It then fell back to pigz-style I/O pipelining, which overlaps file reads
and so does nothing for a buffer already in memory.
This implementation fixes both. Blocks are a fixed ZMAT_BLOCKSIZE (4 MB) with
only the remainder short, workers pull indices from a shared counter so a
slow-compressing block cannot starve the pool, and each block is closed with
Z_FULL_FLUSH -- which ends the block *and* resets the LZ77 window, making it
independently inflatable while the concatenation remains an ordinary zlib or
gzip stream. Z_FINISH would have set BFINAL and stopped every reader at the
first block. The block offsets, free to record at compression time, come back
through a new zmat_run_indexed() so a later decode can start at block N;
zmat_run keeps its signature and its bytes.
pthreads, not OpenMP. -pthread is already linked for blosc2, whereas libgomp
would be a new hard runtime dependency on a toolbox shipping prebuilt binaries
for eight platform/interpreter combinations -- and -static-libgcc, already set
in src/Makefile, does not cover it. Worse, MATLAB ships libiomp5 and its MKL
BLAS links it, so a libgomp-linked MEX would put a second OpenMP runtime in the
process. The workload needs none of what OpenMP provides: N independent
deflate() calls over disjoint slices, then join. The built MEX links libc,
libmex, libmx and libz, with zero GOMP symbols.
67 MB of poorly-compressible uint16, C harness, system zlib 1.2.11
deflate inflate
serial 32.2 MB/s 249.7 MB/s
nthread=4 115.8 MB/s 3.6x 1159.3 MB/s 4.6x
nthread=16 357.2 MB/s 11.1x 3915.3 MB/s 15.7x
size penalty +0.0015% (zlib), and the stream stays readable by plain
zlib.decompress and gzip -dc at every thread count.
Two properties the tests pin down. The blocked construction is opt-in via an
explicit nthread, because it changes the compressed bytes: gating on nthread>1
would make nthread=1 and nthread=2 disagree byte for byte, which breaks content
addressing of the payload, while gating on something always true would change
the output of every existing caller. zmat.m therefore defaults nthread to 0,
meaning "never asked", and a default zmat(x,1,'zlib') call is byte-identical to
unmodified master built with the same flags (verified: same sha256 on 16 MB).
And a block index that does not describe the stream is rejected rather than
trusted, falling back to a serial inflate, so a stale index costs time but
never correctness.
The output is also byte-identical to jdata.zlibmt's pure-Python
implementation -- same 57,613,937 bytes and sha256 on a 67 MB payload, with
each able to read the other's index -- so the same content lands under the same
content-addressed name whether MATLAB, Octave or Python wrote it.
Verified: 44 C correctness checks across sizes 1 B to 17 MB spanning the block
boundary, covering unaware-reader compatibility, indexed parallel inflate,
corrupt-index fallback and thread-count independence; clean under -Wall -Wextra
in the zlib, miniz (NO_ZLIB) and NO_PTHREAD configurations; and zmat's own test
suite output is unchanged from master apart from stack-trace line numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
-pthread now reaches both the compile and the link line unconditionally, rather
than arriving only as a side effect of blosc2 being enabled. It has to be
appended to LIBZLIB after the codec conditionals, because the HAVE_ZLIB!=yes
branch clears that variable and would silently drop an earlier append -- which
is the default miniz build, i.e. the one that matters. CMake gains
Threads::Threads with THREADS_PREFER_PTHREAD_FLAG; python/setup.py already
linked pthread.
ZMAT_DEFAULT_NTHREAD, 8, is applied when the caller does not name a thread
count, and is overridable at build time with -DZMAT_DEFAULT_NTHREAD=N. Since
the blocked construction changes the compressed bytes, this changes the default
output of zlib and gzip for inputs above the 4 MB block size. A negative
nthread is the way back to the historical single-stream bytes, and it is exact:
zmat(x, 1, 'zlib', 'nthread', -1) reproduces the sha256 that the currently
shipped private/zipmat.mexa64 produces on a 16 MB array.
16 MB uint16, MATLAB R2019b, miniz build
default (no nthread) 110.6 MB/s 4 blocks sha f33f69a3...
nthread=8 110.3 MB/s 4 blocks sha f33f69a3...
nthread=1 31.6 MB/s 4 blocks sha f33f69a3...
nthread=-1 32.4 MB/s serial sha 2ac90aee... (shipped bytes)
nthread=1 producing the same bytes as nthread=8 while running serially is the
property worth keeping: the output is a function of the data, level and block
size only, so a payload can be content-addressed without pinning the thread
count of whoever compressed it.
For the record, since it came up: blosc2's own default is a single thread
(g_nthreads = 1 in src/blosc2/blosc/blosc2.c), and zmat has always passed 1
unless asked otherwise. The 4 in circulation is jsonlab's jsave.m, not a blosc2
or zmat default. blosc2's threading is left as it was.
zmat's test suite is unchanged against the shipped binary: same 245 lines, the
same 5 pre-existing failures, no new ones. Those failures are small-input zlib
golden-byte cases that sit below the block size and so never reach the threaded
path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Auditing the bindings and CI for the pthread dependency turned up one build
that my previous commit would have broken outright and two that would have
linked only by luck.
The Windows wheels are the broken one. build_windows_wheel.yml runs
"python -m build --wheel" on windows-2022, which uses MSVC -- confirmed by
setup.py's "/O2" branch -- and MSVC has no pthreads at all. setup.py links no
pthread there and does not define NO_PTHREAD, so zmatlib.c's pthread_create
would simply have failed to resolve. Rather than switch threading off on that
toolchain, or make it conditional on blosc2's vendored win32 shim being
compiled in, the six primitives this codec actually needs are now behind
zmat_thread_t / zmat_mutex_t, with a Win32 backend (CreateThread,
CRITICAL_SECTION) selected on _MSC_VER. The worker is stashed in the pool so
the Win32 entry point can dispatch without a per-thread allocation. Nothing
else needs to change: Win32 threads live in kernel32, which is always linked,
so setup.py and python/Makefile stay as they are.
The two lucky ones both concern -pthread going missing from a flag set that
gets rebuilt rather than appended to:
- src/Makefile replaces CPPOPT wholesale in its Windows branch, so appending
$(PTHREADOPT) at the definition site silently dropped it there. It is now
appended after the platform conditionals, matching the fix already applied
to LIBZLIB for the HAVE_ZLIB!=yes branch.
- src/compilezmat.m, the MATLAB-side builder, passed no -pthread on either
the compile or the link line, in either its MATLAB or its Octave branch. It
now does, skipped under ispc where the Win32 path applies instead.
Everything else was already covered and is left alone: example/c/Makefile and
example/f90/Makefile link -lpthread; python/setup.py appends pthread on every
non-Windows platform; python/Makefile does the same for Linux, and macOS needs
nothing because pthreads is in libSystem; run_test.yml installs
mingw-w64-x86_64-winpthreads-git for the MinGW builds and passes -lwinpthread
or -lpthread explicitly on Windows and macOS; and the lib target builds a
static archive where thread linkage is the consumer's job.
Verified: all of make lib, dll, mex, oct and example/c build; the C correctness
suite still passes 44/44; zmat's MATLAB suite is byte-identical to the shipped
binary with no new failures; the Octave binding round-trips at nthread 8, 1 and
-1; and the MSVC branch is syntax-clean under a stubbed windows.h, since no
MSVC is available here to compile it for real. That last one is the weak spot
in this commit -- the Win32 path has been compiled but never run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docs claimed the output "is a pure function of (data, level, blocksize) and is byte-identical however many threads produced it". True of the blocked construction -- nthread of 1, 8 and 32 give the identical bytes -- but it read as if it covered the serial path too, which it cannot: a negative nthread selects a different construction with a different length. Both are now stated side by side, along with the ratio cost of resetting the window at every block boundary, which ranges from +0.000% on 32 MB of random uint16 to +21% on a long monotonic counter whose redundancy spans the block size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pyzmat.c called zmat_run everywhere, so the Python module could never reach the
threaded path no matter what nthread it was handed -- the nthread argument in
zmat/__init__.py was accepted and then quietly ignored for zlib and gzip. Only
the MATLAB/Octave MEX was wired. My earlier claim that the bindings were covered
was about pthread linkage, not about which entry point they call.
zmat() now routes through zmat_run_indexed and gains the two arguments needed to
use it: offsets, a block index from a previous compression that lets zlib and
gzip inflate concurrently, and return_offsets, which switches the return from
bytes to (bytes, index). compress() and decompress() are routed through it too,
so they pick up the build-time default thread count; encode() and decode()
default to base64, which the threading gate excludes, so they still call
zmat_run.
nthread now defaults to 0 rather than 1, matching zmat.m. Under the new
semantics 1 means "explicitly one thread", which selects the blocked
construction and so changes the output bytes; 0 means "unspecified" and lets the
library decide. Leaving the old default of 1 in place would have silently
switched every existing caller to blocked output while giving them none of the
speed.
16.8 MB uint16, python3.10
default / nthread=8 122.6 MB/s sha d9334b6d...
nthread=1 33.5 MB/s sha d9334b6d... same bytes, one thread
nthread=-1 33.7 MB/s sha c0f65053... historical stream
inflate, serial 258.8 MB/s
inflate, 8 + index 1038.9 MB/s 4.0x
Also fixes a build bug found on the way. ensure_csrc() returned early whenever
csrc/src/zmatlib.c already existed, so a populated csrc was never refreshed from
../src: editing the C sources and rebuilding the wheel silently produced a
module from the previous sources, which is exactly how this change first
appeared to fail with an undefined zmat_run_indexed. A clean checkout is
unaffected -- csrc is untracked and gets populated on first build -- but any
working tree that has built before was affected. When the parent tree is present
it is now treated as the authority. Note that copy2 preserves mtimes, so a stale
build/ directory can still skip the recompile; rm -rf build after changing ../src.
Verified: the module builds clean, links pthread with zero GOMP symbols, the 64
existing tests pass, plain zlib.decompress reads the threaded output, a corrupt
index falls back to correct bytes, and base64 encode/decode are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fortran90/zmatlib.f90 is a pure bind(C) interface block, so wiring it meant two
things.
First, the simplified C helpers it declares. zmat_encode and zmat_decode called
zmat_run with a bare literal for iscompress, which leaves the packed thread byte
at 0; routing them through zmat_run_indexed with a null index means that 0 is
read as "unspecified" and zlib/gzip pick up ZMAT_DEFAULT_NTHREAD. Plain C callers
of those two helpers get the same benefit. Everything else about them is
unchanged.
Second, a declaration for zmat_run_indexed itself, so Fortran callers can hand
back a block index and inflate concurrently. offsets is a size_t** on the C side,
which maps to an intent(inout) type(c_ptr): set it to C_NULL_PTR to decline the
index, or pass the one from a previous compression. It is malloc'ed by the
library and freed with zmat_free.
Exercised from a real Fortran program on 16 MiB of the worst-ratio payload, not
just compiled:
serial (nthread=-1) 113760 bytes, no index
threaded (nthread=8) 137868 bytes, 4 blocks, 10-entry index
indexed inflate 16777216 bytes, roundtrip ok
index[0..3] 2, 0, 34467, 4194304
Those match the C measurements exactly, and the first block starting at byte 2
is the zlib header, as expected. The pre-existing example/f90 still builds and
prints the same output.
One wrinkle worth recording for anyone packing the flags by hand: the thread
count is byte 1 of the level argument, so nthread=-1 is z'0000FF01' and not
z'FFFF0001' -- getting that wrong silently selects the default instead of the
serial path, which is how the first run of the test above appeared to produce a
blocked stream for its serial case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All five failures the suite has been carrying predate this branch -- master
built with the same flags produces byte-identical output -- and all five turn
out to be expectations that drifted from the code, not defects.
Four are the zlib header's FLG byte, and only in the isminiz arm of each pair;
the system-zlib arm already had the right values. FLG carries FLEVEL, which
encodes the compression level, so it cannot be the same for every level -- and
the giveaway is that level=9 and level=2.6 were given identical byte arrays,
which no conforming deflate can produce. The recorded values all had FLG=1
(FLEVEL=0), from a time when the header was written level-independently.
Measured now, both backends and both interpreters agree:
scalar, array 120 156 FLEVEL=2 (default)
level=9 120 218 FLEVEL=3
level=2.6 120 94 FLEVEL=1
The fifth is blosc2zstd (typesize=2), where the vendored blosc2 changed one bit
of its own container flags byte, 0x97 to 0x87. zmat does not write that header.
Both typesizes still round-trip, so this is drift in the golden rather than a
functional break.
Separately, sprand was the suite's only random input and was never seeded, so
the compressed size it reports differed on every run -- master alone gave 14896
then 14913 on two consecutive runs. That made any run-to-run comparison noisy
enough to hide a real change, which is how it first looked like this branch had
altered something. Seeded, two Octave runs are now byte-identical.
MATLAB and Octave both report zero failures.
Also promotes the block-parallel correctness harness out of scratch and into
test/test_zlibmt.c, with build lines for both backends in its header. It covers
sizes from 1 byte to 17 MB spanning the block boundary: unaware-reader
compatibility, indexed parallel inflate, corrupt-index fallback, and
thread-count independence. Both backends pass. It skips the gzip unaware-reader
check under miniz, whose inflateInit2 rejects window bits 15|32 -- that check
cannot run there for any gzip stream, zmat's or otherwise, so reporting it as a
failure said nothing about the data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d header Rebasing onto NeuroJSON/zmat brought in nine commits that overlap this work -- multi-threading for xz, lzip, zstd and lzma, per-codec shuffle, an info= form for the Python wrapper, and a generated single-header build. Three things needed deciding rather than merging. The thread-count default. Upstream set zmat.m to 4 for its MT codecs; this branch had used 0 to mean "caller said nothing" so zlib/gzip could tell opt-in from default. Both now resolve to 8, so lzip/lzma/xz/zstd/blosc2 get 8 rather than 4 and zlib/gzip get 8 rather than 1. A negative nthread remains the way back to zlib/gzip's historical single-stream bytes; that escape hatch is the only thing the 0 sentinel was protecting, and an explicit default states the intent more plainly. The version. Upstream already shipped 1.1.0 and its commit says "start 1.2.preview", so the bump here is to 1.2.0 -- git dropped this branch's own 1.1.0 bump during the rebase as already-applied upstream. The Python signature. Upstream's info= and this branch's offsets/return_offsets both survive, with offsets threaded through the info-dict decompress path as well as the plain one. return_offsets cannot be combined with info, since the info forms already return a tuple, so that combination now raises rather than silently ignoring the request. Also regenerates include/zmat.h. Upstream added it as a generated single-header library, but it was generated before this work, so it carried neither zmat_run_indexed nor the parallel implementation. Rebuilt with "make -C src header"; it now compiles standalone and threads correctly. Verified against the merged tree: MATLAB 253 lines and Octave 308 lines with zero failures, the C block-parallel harness passing on both the zlib and miniz backends, the header-only library round-tripping 16 MiB across 4 blocks, and the Python module reporting 1.2.0 with lzma, zstd, lz4 and blosc2zstd round-tripping alongside the new zlib/gzip path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Block-parallel zlib and gzip, plus a block index so the result can be inflated
in parallel too.
DEFLATE has no framing: a decoder cannot locate block N without inflating
everything before it, and back-references reach arbitrarily far into the 32 KB
window. Two things make a parallel codec possible anyway.
Z_FULL_FLUSHends ablock and resets the window, so each block becomes independently inflatable
while the concatenation stays an ordinary zlib/gzip stream any decompressor can
read start to finish. And the block offsets, free to record at compression
time, are handed back so a later decode can start at block N.
Why pthreads and not OpenMP
-pthreadwas already linked for blosc2, whereas libgomp would be a new hardruntime dependency for a project shipping prebuilt binaries across eight
platform/interpreter combinations — and
-static-libgcc, already set insrc/Makefile, does not cover it. Worse, MATLAB shipslibiomp5and its MKLBLAS links it, so a libgomp-linked MEX would put a second OpenMP runtime in the
process. The workload needs none of what OpenMP offers: N independent
deflate()calls over disjoint slices, then join.MSVC has no pthreads at all and is what builds the Windows wheels, so the six
primitives used here sit behind a small abstraction with a Win32 backend
(
CreateThread,CRITICAL_SECTION) selected on_MSC_VER.An earlier attempt, and why it concluded otherwise
The
pzlibbranch ends with "[deadend] multi-threaded zlib/gzip offers nospeedup for memory-based buffer". That was an artifact of two bugs rather than
a property of the problem: the block count was capped at the thread count while
the block size stayed fixed at 128 KB, so the final block received 88–98% of the
input and the maximum achievable speedup was 1.02–1.13x whatever the thread
count; and nothing emitted
Z_FULL_FLUSHor recorded offsets, so the decoderwas attempting to inflate a slice of an ordinary deflate stream, which is not
possible, and fell back to pigz-style I/O pipelining that does nothing for a
buffer already in memory.
Behaviour changes
window at each boundary is what makes a block independently inflatable, and it
costs ratio: +0.000% on incompressible data, about +21% on data whose
redundancy spans a block (the absolute cost is ~8 KB per boundary, so it only
looks large when the compressed output is tiny).
nthreadnegative reproducesthe previous bytes exactly — verified against the currently shipped
private/zipmat.mexa64.nthreadnow defaults to 8 rather than 4, so lzip/lzma/xz/zstd/blosc2 get8 and zlib/gzip get 8 instead of 1.
Within the blocked construction the output is a pure function of (data, level,
blocksize):
nthreadof 1, 8 or 32 give identical bytes, so a payload can becontent-addressed without pinning the thread count of whoever compressed it. It
is also byte-identical to
jdata.zlibmt's pure-Python implementation — same57,613,937 bytes and sha256 on a 67 MB payload — so the same content lands under
the same name whether MATLAB, Octave or Python wrote it.
Also in here
offsets/return_offsets), Fortran, and the Czmat_encode/zmat_decodehelpers.zmat_runitself is unchanged and stays serial.-pthreadon compile and link insrc/Makefile(appended after the codecconditionals, since the
HAVE_ZLIB!=yesbranch clearsLIBZLIB), CMake, andsrc/compilezmat.m.include/zmat.hregenerated so the single-header build carries the new code.test/test_zlibmt.c: sizes from 1 byte to 17 MB spanning the block boundary,covering unaware-reader compatibility, indexed parallel inflate,
corrupt-index fallback and thread-count independence.
run_zmat_test.m. Four were the zlibheader's FLG byte in the
isminizarm — FLG carries FLEVEL, so it tracks thecompression level, and the recorded values had
level=9andlevel=2.6expecting identical bytes. The fifth was a blosc2 container flags bit that
drifted with the vendored blosc2. Also seeded
sprand, the suite's onlyrandom input, which made the reported size differ on every run.
Verification
MATLAB 253 lines and Octave 308 lines with zero failures; Python 64 tests OK on
8 versions (3.7–3.14) on Windows CI under MSVC; the C harness passing on both
the zlib and miniz backends; the header-only library round-tripping standalone;
make lib/dll/mex/oct,example/candexample/f90all building and running.One gap worth naming: the Win32 threading backend is exercised on Linux by
backing its primitives with pthreads (7.15x scaling, all correctness checks),
and MSVC compiles it in CI with no new warnings — but it has not been run on a
real Windows machine.