Repository navigation
transparent gzip compression in urllib #43521
Description
| BPO | 1508475 |
|---|---|
| Nosy | @rhettinger, @jcea, @orsenthil, @vstinner, @merwok, @berkerpeksag, @vadmium, @JimJJewett, @serhiy-storchaka, @demianbrecht |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2006-06-19.08:59:09.000>
labels = ['type-feature', 'library']
title = 'transparent gzip compression in urllib'
updated_at = <Date 2015-03-07.02:32:39.334>
user = 'https://bugs.python.org/antialize'bugs.python.org fields:
activity = <Date 2015-03-07.02:32:39.334>
actor = 'demian.brecht'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2006-06-19.08:59:09.000>
creator = 'antialize'
dependencies = []
files = ['7335', '19811', '34177']
hgrepos = []
issue_num = 1508475
keywords = ['patch']
message_count = 23.0
messages = ['50500', '50501', '114671', '114725', '114726', '122342', '122343', '122351', '158315', '158380', '158400', '160355', '160384', '163925', '163935', '211893', '212252', '212473', '213982', '226258', '226399', '234363', '236013']
nosy_count = 18.0
nosy_names = ['rhettinger', 'jjlee', 'jcea', 'orsenthil', 'jerub', 'vstinner', 'antialize', 'ruseel', 'nadeem.vawda', 'thomaspinckney3', 'eric.araujo', 'abacabadabacaba', 'jcon', 'berker.peksag', 'martin.panter', 'Jim.Jewett', 'serhiy.storchaka', 'demian.brecht']
pr_nums = []
priority = 'high'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1508475'
versions = ['Python 3.5']Linked PRs
Metadata
Metadata
Assignees
Projects
- Status
- Status
Activity
[-]transparent gzip compression in liburl2[/-][+]transparent gzip compression in urllib[/+][-]transparent gzip compression in liburl2[/-][+]transparent gzip compression in urllib[/+]orsenthil commentedon Nov 25, 2010
The transparent gzip Content-Encoding support should be done at the
http.client level code.
Before adding this feature, a question needs to be sorted out.
If we support the transparent gzip and wrap the file pointer to a
GzipFile filepointer, should reset the Content-Length value?
What if a user of urllib is relying on the Content-Length of response
to do something further?
I observed that google-chrome returns the uncompressed output (which
is correct for a browser), but has the Content-Length set the
compressed output length.
orsenthil commentedon Nov 25, 2010
Patch for py3k.
merwok commentedon Nov 25, 2010
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13
serhiy-storchaka commentedon Apr 15, 2012
What if the gzip module is not available?
I think, with transparent decompression should delete headers Content-Encoding (to free the user from re-decompression) and Content-Length (which is wrong).
orsenthil commentedon Apr 15, 2012
In that case, transparent decompression should not be available. (
Request header should not be sent and response wont be compressed).
serhiy-storchaka commentedon Apr 16, 2012
The patch for py3k also has the disadvantage that the content is decoded even if the user has defined a Content-Encoding and he is going to process compressed response himself.
thomaspinckney3 commentedon May 10, 2012
What if this gzip decompression was optional and controlled via a flag or handler instead of making it automagic?
It's not entirely trivial to implement so it is nice to have the option of this happening automatically if one wishes.
Then, the caller would be aware that Content-length / Accept-encoding / Content-encoding etc have been modified iff they requested gzip decompression.
merwok commentedon May 10, 2012
Enabled by default with a knob to turn it off sounds good. Maybe the original headers could be preserved in some object.
serhiy-storchaka commentedon Jun 25, 2012
The first step is to answer on the fundamental question: on what level transparent decompression will work? On http.client level or on urllib level? Patch for first case will be much more difficult, but will benefit from compression in other http-based protocols.
orsenthil commentedon Jun 25, 2012
I think, the transparent compression should work at http.client level. I also agree with other points made by Serhiy:
transparent decompression should delete headers Content-Encoding and Content-Length (this is as per RFC too)
Should not do another compression if the user has a explicit specified intent of using Content-Encoding: gzip and is ready to do decompression himself.
This transparent compression/decompression would require the availability gzip module, if not then the feature may be disabled and normal request-response cycle would proceed.
I think, having it 'ON' with a flag to switch 'OFF' would be more desirable than having this feature via Handler. The reason being it can help in performance of any requests on servers that support it and browsers have adopted similar approach too.
vstinner commentedon Feb 21, 2014
I updated bpo-1508475.diff for Python 3.4 and removed the change in urllib: http_client_gzip.patch. The patch only changes http.client to support server sending gzip data.
For example, the new python.org website serves gzip data even if the Accept-Encoding header is not sent by the client: see the issue bpo-20719.
vadmium commentedon Feb 26, 2014
I have code that already handles an “gzip” encoded response from urlopen(). All three patches leave the Content-Encoding header intact, so I suspect my code would try to decompress the body a second time. Deleting this header (as already suggested) would work for me.
rhettinger commentedon Feb 28, 2014
Victor, the patch looks good and would be a welcome enhancement.
There should be an option for turning this on and off (perhaps, I want the zipped content and want to unzip later or in a different thread).
Consider adding support for "deflate" as well.
jimjjewett commentedon Mar 18, 2014
This is an enhancement, so I am changing the affected version from 3.3 to 3.5.
It is python-only, which works well with the cheeseshop.
That said, the patch is truly short; if that is really sufficient, it could almost go into the documentation as a recipe. But I would prefer some more assurances that it actually does work; a quick skim suggests that it relies on a superclass happening to implement read via readinto.
Needs tests and documentation change.
vadmium commentedon Sep 2, 2014
I think the patch is indeed a bit short, for instannce it looks like calling read() without a size limit could bypass the decoding.
Also, I wonder if Content-Encoding handling is better done at a higher level. What if someone wants to download a *.tar.gz file? They may not expect the tar file to be transparently decompressed. And I suspect this would blow up if you tried a partial range request.
Transfer-Encoding is meant to be the proper way to transparently compress HTTP messages at a low level, but it doesn’t seem to be used as much in the real world.
vadmium commentedon Sep 5, 2014
Related: bpo-1243678, which includes a patch for “httplib” (now known as “http.client”?). That patch looks like it sets Accept-Encoding and decodes according to Content-Encoding. However I suspect it is also trying to be too “transparent” at the wrong level and would have many of the problems already mentioned here.
vadmium commentedon Jan 20, 2015
The Lib/xmlrpc/client.py file appears to already support compression using “Content-Encoding: gzip”. Perhaps it could be leveraged for any work on this issue.
vadmium commentedon Feb 15, 2015
I suggest resolving bpo-15955 first, then the GzipFile API could be used without fear of decompression bombs.
soreavis commentedon Jul 19, 2026
Serhiy, when this came up for bulk-closing you said transparent gzip was a nice-to-have "if anybody provide PRs" — though not easy. Fair enough on that — it never really stalled on code, it stalled on one design call: opt-in handler vs default-on at http.client. That call's yours; here's one concrete answer to it: #154197.
I went opt-in — a HTTPGzipHandler that's not in build_opener()'s defaults, so urlopen() is unchanged unless you ask for it. That sidesteps Martin's 2014 footgun, where default-on would silently decompress a .tar.gz and break Range requests. It decodes only when it added the Accept-Encoding itself, and strips Content-Encoding/Content-Length.
Senthil preferred default-on with a flag; I can flip it if you'd rather — the mechanism's identical. gzip only for now, deflate a follow-up.
Merge remote-tracking branch 'upstream/main' into pythongh-43521-gzip…
Merge remote-tracking branch 'upstream/main' into pythongh-43521-gzip…
pythongh-43521: Skip the HTTPGzipHandler tests when zlib is unavailable
soreavis commentedon Oct 27, 2026
@orsenthil, #154197 has been open three months without a review. It's opt-in:
build_opener()doesn't installHTTPGzipHandler, sourlopen()behaves as before. It follows your 2012 list on headers and caller intent: Content-Encoding and Content-Length are dropped after decoding, and a request that already sets Accept-Encoding is left alone. It merges cleanly with main and the tests pass. You preferred default-on over a handler; I went opt-in because of Martin's 2014 .tar.gz and Range point. Is opt-in OK, or should I flip it?orsenthil commentedon Oct 27, 2026
@soreavis , opt-in better and conservative for this long pending request. I will review this.
Merge branch 'main' into pythongh-43521-gzip-handler
pythongh-43521: Decode gzip responses with gzip.GzipFile