Repository navigation
fix(recorder): honor Content-Encoding when replaying text record files - #847
Shellghostt wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5989970. Configure here.
|
Fixed in 386c7e4 — good catch, verified before fixing: a text body replayed with The regression test now carries a deliberately wrong |
| # Bodies are stored in record files decoded, so a | ||
| # Content-Encoding header left in the file (recorded by an | ||
| # older version, or preserved via keep_headers) no longer | ||
| # describes the body and makes clients fail to decode it | ||
| # on replay. The Content-Length recorded alongside it | ||
| # measures the encoded body, so it is dropped as well; | ||
| # urllib3 otherwise enforces it against the decoded body. | ||
| # Encoded (bytes) bodies keep both headers. |
There was a problem hiding this comment.
This is unwise. We shouldn't start randomly ignoring headers that have explicitly been defined.
There was a problem hiding this comment.
Good point — changed. _add_from_file no longer removes any headers. When a text body declares Content-Encoding: gzip/x-gzip/deflate, it now compresses the stored text to that coding and corrects a declared Content-Length to the encoded size, so the explicit headers are honored and the YAML still keeps the human-readable decoded body. Bytes bodies are unchanged. Pushed as ee9a8e1 with tests for gzip text (including a stale length), deflate text, and gzip bytes. Please take another look.
There was a problem hiding this comment.
Closing the loop on this one, with the measurements behind the change.
The fix now keeps every header that was explicitly declared and makes the stored body match it, instead of dropping the header. The reason that is the right direction is that _recorder.py stores requests_response.text, which is the decoded body, and copies the response headers unchanged. A record file therefore says Content-Encoding: gzip with a Content-Length measured on the compressed response, while holding plain text. The header was not wrong, the body was in its decoded form, so removing the header is what broke the replay.
What _add_from_file does now: a text body whose record file declares gzip, x-gzip or deflate is compressed to that coding, and a declared Content-Length is rewritten to the size of the encoded body. Bytes bodies are already encoded and pass through untouched, and any other coding is left alone.
Measured on the same nine record files, before and after:
| case | upstream master e68019c |
with ee9a8e1 |
|---|---|---|
| gzip text + stale length | ContentDecodingError: incorrect header check |
200, body correct, length corrected 5 to 36 |
| gzip text, no length, auto=True | ContentDecodingError |
200, body correct, length 36 |
| gzip text, stale length, auto=True | ContentDecodingError |
200, body correct, length 36 |
| x-gzip text | ContentDecodingError |
200, body correct |
| deflate text | ContentDecodingError: invalid distance too far back |
200, body correct |
| gzip bytes body | 200 | 200, unchanged |
br, identity, no encoding |
200 | 200, unchanged |
The length is only rewritten when the record file declared one, so an explicit length is corrected rather than discarded. When the record file declared none and auto_calculate_content_length is set, get_response adds a length measured on the encoded body, which is the compressed size, so that path is right as well.
This also covers the Cursor Bugbot finding on the original commit, where the recorded length described the compressed body while the decoded body was served, which broke the replay with IncompleteRead on urllib3 2.x.
The branch is ee9a8e1. The full suite is 262 passed, and black, isort and flake8 are clean on both changed files.
386c7e4 to
ee9a8e1
Compare
| headers = None | ||
|
|
||
| body = rsp["body"] | ||
| if headers is not None: |
There was a problem hiding this comment.
doesn't the _encode_recorded_text_body already handle this scenario
There was a problem hiding this comment.
Right, it does. It returns early when headers is None or the body is not text, so the if headers is not None wrapper was doing nothing the helper does not already do. Removed it in d4823a3, so _add_from_file now calls the helper unconditionally. I also rebased the branch onto master 7506408, which clears the conflict with the Python range change in responses/tests/test_recorder.py, so those two imports now sit together. 262 tests still pass and black, isort and flake8 are clean.
Record files store text bodies decoded, so a declared Content-Encoding no longer matched the body and replay failed to decode it. Instead of dropping the explicit header, compress the stored text back to the declared gzip/x-gzip/deflate coding and correct a declared Content-Length to the encoded size. Bytes bodies pass through unchanged.
657d3c8 to
d4823a3
Compare

Fixes #724
Summary
Replaying a record file that still carries
Content-Encoding: gzipover adecoded text body fails with
ContentDecodingError, because_add_from_fileregistered the header while the stored body is plain text.
Change
responses/__init__.py(_add_from_file): when a record file declaresgzip,x-gzipordeflatefor a text body, compress the stored textto that coding before replay and correct a declared
Content-Lengthto theencoded size. The explicit headers stay in effect instead of being dropped,
while the YAML still keeps the human-readable decoded body.
responses/tests/test_recorder.py: regression test covering gzip text (witha stale
Content-Length), deflate text, and gzip bytes bodies.Tests
pytest responses/tests/test_recorder.py responses/tests/test_responses.py— 179 passed.
black --check --line-length=88,isort --check-only --force-single-line-imports --profile black,and
flake8 --max-line-length 100 --max-doc-length 120on both changedfiles — clean.
content-encoding: gzip) replays200with the expected JSON.