Skip to content

fix(recorder): honor Content-Encoding when replaying text record files - #847

Open
Shellghostt wants to merge 2 commits into
getsentry:masterfrom
Shellghostt:fix-recorder-content-encoding-replay
Open

Shellghostt wants to merge 2 commits into
getsentry:masterfrom
Shellghostt:fix-recorder-content-encoding-replay

Conversation

@Shellghostt

@Shellghostt Shellghostt commented Nov 8, 2026 •

Copy link
Copy Markdown

Fixes #724

Summary

Replaying a record file that still carries Content-Encoding: gzip over a
decoded text body fails with ContentDecodingError, because _add_from_file
registered the header while the stored body is plain text.

Change

  • responses/__init__.py (_add_from_file): when a record file declares
    gzip, x-gzip or deflate for a text body, compress the stored text
    to that coding before replay and correct a declared Content-Length to the
    encoded size. The explicit headers stay in effect instead of being dropped,
    while the YAML still keeps the human-readable decoded body.
  • Bytes bodies are already encoded and pass through unchanged.
  • responses/tests/test_recorder.py: regression test covering gzip text (with
    a stale Content-Length), deflate text, and gzip bytes bodies.

Tests

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5989970. Configure here.

Comment thread responses/__init__.py Resolved
@Shellghostt

Copy link
Copy Markdown
Author

Fixed in 386c7e4 — good catch, verified before fixing: a text body replayed with Content-Length: 5 against an 18-byte body fails with IncompleteRead(18 bytes read, -13 more expected). When _add_from_file strips Content-Encoding for a string body, it now also strips the Content-Length recorded alongside it (it measured the encoded body). Bytes bodies keep both headers and still decode.

The regression test now carries a deliberately wrong Content-Length: 5 on the text response and asserts the stale value is gone on replay. test_recorder.py (12) and test_responses.py (167) pass, and the standalone legacy-file reproduction returns 200 {"hello": "world"}.

Comment thread responses/__init__.py Outdated
Comment on lines +904 to +911
# Bodies are stored in record files decoded, so a
# Content-Encoding header left in the file (recorded by an
# older version, or preserved via keep_headers) no longer
# describes the body and makes clients fail to decode it
# on replay. The Content-Length recorded alongside it
# measures the encoded body, so it is dropped as well;
# urllib3 otherwise enforces it against the decoded body.
# Encoded (bytes) bodies keep both headers.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unwise. We shouldn't start randomly ignoring headers that have explicitly been defined.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — changed. _add_from_file no longer removes any headers. When a text body declares Content-Encoding: gzip/x-gzip/deflate, it now compresses the stored text to that coding and corrects a declared Content-Length to the encoded size, so the explicit headers are honored and the YAML still keeps the human-readable decoded body. Bytes bodies are unchanged. Pushed as ee9a8e1 with tests for gzip text (including a stale length), deflate text, and gzip bytes. Please take another look.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closing the loop on this one, with the measurements behind the change.

The fix now keeps every header that was explicitly declared and makes the stored body match it, instead of dropping the header. The reason that is the right direction is that _recorder.py stores requests_response.text, which is the decoded body, and copies the response headers unchanged. A record file therefore says Content-Encoding: gzip with a Content-Length measured on the compressed response, while holding plain text. The header was not wrong, the body was in its decoded form, so removing the header is what broke the replay.

What _add_from_file does now: a text body whose record file declares gzip, x-gzip or deflate is compressed to that coding, and a declared Content-Length is rewritten to the size of the encoded body. Bytes bodies are already encoded and pass through untouched, and any other coding is left alone.

Measured on the same nine record files, before and after:

case upstream master e68019c with ee9a8e1
gzip text + stale length ContentDecodingError: incorrect header check 200, body correct, length corrected 5 to 36
gzip text, no length, auto=True ContentDecodingError 200, body correct, length 36
gzip text, stale length, auto=True ContentDecodingError 200, body correct, length 36
x-gzip text ContentDecodingError 200, body correct
deflate text ContentDecodingError: invalid distance too far back 200, body correct
gzip bytes body 200 200, unchanged
br, identity, no encoding 200 200, unchanged

The length is only rewritten when the record file declared one, so an explicit length is corrected rather than discarded. When the record file declared none and auto_calculate_content_length is set, get_response adds a length measured on the encoded body, which is the compressed size, so that path is right as well.

This also covers the Cursor Bugbot finding on the original commit, where the recorded length described the compressed body while the decoded body was served, which broke the replay with IncompleteRead on urllib3 2.x.

The branch is ee9a8e1. The full suite is 262 passed, and black, isort and flake8 are clean on both changed files.

@Shellghostt
Shellghostt force-pushed the fix-recorder-content-encoding-replay branch from 386c7e4 to ee9a8e1 Compare November 11, 2026 00:17
@Shellghostt Shellghostt changed the title fix(recorder): strip Content-Encoding when replaying text record files fix(recorder): honor Content-Encoding when replaying text record files Nov 11, 2026
Comment thread responses/__init__.py Outdated
headers = None

body = rsp["body"]
if headers is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't the _encode_recorded_text_body already handle this scenario

@Shellghostt Shellghostt Nov 16, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it does. It returns early when headers is None or the body is not text, so the if headers is not None wrapper was doing nothing the helper does not already do. Removed it in d4823a3, so _add_from_file now calls the helper unconditionally. I also rebased the branch onto master 7506408, which clears the conflict with the Python range change in responses/tests/test_recorder.py, so those two imports now sit together. 262 tests still pass and black, isort and flake8 are clean.

@pbhandari
pbhandari dismissed their stale review November 16, 2026 20:02

change was implemented

Record files store text bodies decoded, so a declared Content-Encoding no longer matched the body and replay failed to decode it. Instead of dropping the explicit header, compress the stored text back to the declared gzip/x-gzip/deflate coding and correct a declared Content-Length to the encoded size. Bytes bodies pass through unchanged.
@Shellghostt
Shellghostt force-pushed the fix-recorder-content-encoding-replay branch from 657d3c8 to d4823a3 Compare November 17, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_recorder.record captures header 'content-encoding: gzip' but does not return gzip content on replay

2 participants