EndlessZstdDecompressor.decompress(max_length=N) can return more than N bytes on multi-frame input (regression since 0.19.0) #72

Summary

EndlessZstdDecompressor.decompress documents "max_length: … When it is nonnegative, returns at most max_length bytes of decompressed data." On input made of several concatenated zstd frames, it can return far more than max_length, because the per-call budget is decremented by the cumulative output length instead of the per-frame length. This is a regression introduced by the 0.19.0 pure-Python rewrite; the 0.18.0 C core returned the correct amount.

Reproduction (pyzstd==0.20.0, benign input)

from pyzstd import compress, EndlessZstdDecompressor

def frame(n):                                 # one zstd frame of n identical bytes
    return compress(b"A" * n)

blob = frame(1) + frame(9) + frame(4000)      # three concatenated frames
d = EndlessZstdDecompressor()
out = d.decompress(blob, max_length=10)        # caller asks for <= 10 bytes
print(len(out))                                # -> 4010  (documented cap is 10)

Expected vs actual

• Expected: len(out) <= 10.
• Actual: len(out) == 4010.

Root cause (src/pyzstd/__init__.py, 0.20.0)

out = b""
while True:
    out += self._decompressor.decompress(self._buffer, max_length)
    if self._decompressor.eof:
        self._reset(self._decompressor.unused_data)
        max_length -= len(out)        # BUG: len(out) is cumulative, not this frame's bytes
    else:
        self._buffer = b""
        break
return out

Trace: frame1 → out=1, max_length = 10-1 = 9; frame2 → out=10, max_length = 9-10 = -1; iteration 3 → decompress(frame3, -1), where negative max_length means unlimited, so the whole 4000-byte frame is appended.

The single-frame sibling ZstdDecompressor (and the stdlib layer below) honour max_length correctly; only the EndlessZstdDecompressor multi-frame loop is affected.

Suggested fix (one line)

Decrement by the bytes produced in the current iteration, not the cumulative total:

chunk = self._decompressor.decompress(self._buffer, max_length)
out += chunk
...
max_length -= len(chunk)        # per-frame, not len(out)

With this, the reproduction returns 10 bytes (matching the pre-0.19.0 C core).

Notes

• Affected: 0.19.0, 0.19.1, 0.20.0. Not affected: ≤ 0.18.0 (C core), and the single-frame ZstdDecompressor.
• Correctness bug; the only security-adjacent angle is for callers that rely on max_length as a memory bound against oversized concatenated-frame input. Found via a prove/refute of the documented max_length contract.