Repository navigation
Fix CacheControl caching of consumed mock responses - #846
Open
Aschenbath wants to merge 1 commit into
Open
Aschenbath wants to merge 1 commit into
Aschenbath wants to merge 1 commit into
Conversation
pbhandari
requested changes
Nov 10, 2026
pbhandari
left a comment
Contributor
There was a problem hiding this comment.
Where specifically in the code is the bug? What specifically does this change do to fix it? What other options were considered?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Description
The bug is in the byte-buffer lifetime created by
responses._handle_body(). The old implementation attached anis_closed()function to a plainBytesIO: reaching EOF inread()leftdata.closed == False, and the buffer closed only when urllib3 subsequently calledisclosed().That timing misses CacheControl's cache-write callback.
CacheControlAdapter.build_response()wraps the mocked response's_fpinCallbackFileWrapper. Itsread()reads the underlying body, copies the bytes into its own buffer, then checksfp is Noneorclosedand calls_close()to commit the cache entry. ABytesIOhas nofpattribute, so this check sees its still-falseclosedproperty. urllib3's laterisclosed()probe closes the original buffer after that opportunity has passed. The adapter does run; the body-completion signal arrives too late._BodyBuffer.read()now closes the mock before returning from an unbounded read, or from a nonzero sized read that returns EOF. CacheControl consequently sees closure during its ownread()and invokes its existing callback.isclosed()only reports state. Positive sized reads that return bytes, zero-byte reads, partial consumption, and caller-providedBufferedReaderobjects retain their previous handling. CacheControl is a test-only dependency; production code has no CacheControl-specific branch.The boundary is deliberate: a positive sized read returning the last bytes does not close early. urllib3 must get the subsequent empty read while checking a remaining
Content-Length; closing as soon as the cursor reaches the buffer length can bypass its truncated-body error path. The added streaming tests verify thatChunkedEncodingErrorstill occurs. This change does not expand the existing unbounded-read behavior for a truncatedContent-Length.Alternatives considered:
isclosed()return value still leaves closure after CacheControl's post-read check.Content-Lengthbehavior above.The reproduction supplies
DateandCache-Controlso the response is eligible for caching; the simplified issue example omits them. This identifies the failure mechanism, without claiming which release between 0.12.1 and 0.25.8 introduced it.This review update changes the explanation only. The validation below was recorded for the unchanged code head
7b9749c8; it is not a claim that the full matrix was rerun for this description edit.Related Issues
Fixes #780.
PR checklist
Full pytest on Python 3.12.12: 289 passed, 18 existing deprecation warnings, 99% coverage. All-files pre-commit and all changed-file hooks pass. Both configured mypy commands report the same three existing errors on the exact base and candidate, with no new diagnostic.
Added/updated tests?
Independent Codex review exercised 68 real CacheControl scenarios per environment, including gzip, zero-byte reads, readinto and partial close. All 32 expected cache hits now occur; the unchanged base has none. Existing urllib3 read/read1/stream traces remain unchanged for the checked journeys.
Compatibility checks include requests 2.32.4 / CacheControl 0.14.2 with urllib3 1.26.20 and 2.8.0 on Python 3.12, plus requests 2.30.0 / urllib3 1.25.10 / CacheControl 0.13.0 on Python 3.11. Each of these three cells passes 42 related tests. The minimum-library cell explicitly pins a yanked CacheControl version and needs its legacy six ImportWarnings reported normally instead of treated as errors. Python 3.8 and macOS were not run locally.
Sdist/wheel builds pass. The installed wheel passes all 28 new tests and the independent 68 cache scenarios; all five packaged source/resource files match the checkout. Existing unbounded-read behavior for a truncated Content-Length is not expanded by this change.
AI assistance: OpenAI Codex investigated, implemented and tested the change. A separate Codex agent reviewed the code and independently verified real HTTP-response behavior, compatibility cells and the installed wheel. No human review is claimed.