Skip to content

gh-127753: make the proxies handling of urllib.request cacheable #127767

Open
NewUserHa wants to merge 6 commits into
python:mainfrom
NewUserHa:patch-4
Open

gh-127753: make the proxies handling of urllib.request cacheable #127767
NewUserHa wants to merge 6 commits into
python:mainfrom
NewUserHa:patch-4

Conversation

@NewUserHa

@NewUserHa NewUserHa commented Dec 9, 2024

Copy link
Copy Markdown
Contributor

this code makes proxy_bypass_registry is also cacheable, to avoid repeating read the registry of windows.

and also makes the comment about CVE of getproxies_environment() together and more obviously

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs a blurb entry.

Comment thread Lib/urllib/request.py Outdated
Comment thread Lib/urllib/request.py Outdated
@NewUserHa

Copy link
Copy Markdown
Contributor Author

actually, this structure of these functions feels no straightforward, new comers may do not know if use the original proxy_bypass() or getproxies(),
or use getproxies_environment() and getproxies_registry() and proxy_bypass_registry() and getproxy_bypass_registry() to build its own cacheable proxy_bypass() or getproxies() to avoid repeat reading from os.

But it looks not good to change or delete the original behavior of the original proxy_bypass() or getproxies() functions, so to make new users know how to correctly use and cache it if the users want to use, should re-arrange or re-name these original functions, or just add documents for it?

@ZeroIntensity

Copy link
Copy Markdown
Member

Sure, if the old APIs are too broken to improve, then add something new.

@github-actions

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 20, 2026
@webknjaz

Copy link
Copy Markdown
Member

@NewUserHa hey, are you planning to get back to this PR? It still needs a news entry included.

@NewUserHa

Copy link
Copy Markdown
Contributor Author

@webknjaz Thanks for your comment.

I reviewed it again, and after digging deeper into the implementation, the core issue is that the proxy_bypass(host) function on Windows opens, reads and closes the winreg on every single invocation. which introduces severe I/O during frequent requests.

However, resolving this has a few architectural dilemmas:

  1. The current approch alters the function signature on windows. This introduces inconsistency on other platforms mac/unix
  2. If preserve the original signatures, using functools.lrucache internally seems like the way. Is it able to import functools for windows?
  3. On MacOS it relies on the _scproxy which is IPC but faster than winreg which is disk I/O. Should this PR scope to only Windows

@github-actions github-actions Bot removed the stale Stale PR or inactive for long period of time. label Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants