Skip to content

Limit proxies list to 100 proxies by default - #1

Merged
meznaric merged 3 commits into
mainfrom
limit-proxies-list-by-default
Sep 14, 2026
Merged

meznaric merged 3 commits into
mainfrom
limit-proxies-list-by-default

Conversation

@meznaric

Copy link
Copy Markdown
Contributor

The problem

webshare proxies list --mode=backbone looks like it hangs. It doesn't — it is paginating the entire proxy list.

Three things compound:

  1. --limit defaulted to 0 = "all", so the loop drove Proxies.ListAll through every page via the envelope's next URL, sequentially.
  2. Results were buffered into a slice and only written after the loop finished — no partial output, no progress, nothing on stderr.
  3. A residential backbone plan holds millions of entries at the server's default page size, so that is thousands of round trips before the first byte of output.

Backbone makes it worse: only mode and country_code__in work as filters there (spec/notes/proxy.md), so most ways a user would try to narrow the list are silently ignored while still paying for the full walk.

The change

  • --limit now defaults to 100, matching the existing default on proxies replaced. --limit 0 restores the old fetch-everything behavior.
  • When the limit cuts the list short, the command says so on stderr — so the cap is never silent, and the notice never corrupts a piped address:port:username:password stream.
  • Help text and README point at proxies download for full exports: the server renders the whole list in a single request, which is the right tool for that job.
  • README examples that redirect to a file now pass --limit 0, so they keep producing complete lists.

Behavior

Against a fake API serving a residential-sized list (6M entries, 60/page):

before after
proxies list --mode=backbone no output after 60s+ 100 rows in 8ms
proxies list --mode=backbone --limit 0 same as before same as before

Tests

Three new tests drive a multi-page fake API and assert on the number of pages actually fetched, not just the output:

  • default limit stops after 2 pages of 60 rather than walking all 100
  • --limit 0 still fetches every page
  • a list shorter than the limit is returned whole, with no truncation notice

Verified the cap is what's doing the work: with defaultProxyListLimit set back to 0, TestProxiesListStopsAtDefaultLimit never completes.

Breaking change

Scripts relying on proxies list returning the complete list need --limit 0 (or, better, proxies download). This is worth a minor version bump — proposing v0.2.0.

Vito Meznaric added 3 commits September 14, 2026 15:38
`proxies list` walked every page of the proxy list and buffered the whole
thing before printing a single byte. On a residential backbone plan, where
the list runs to millions of entries at ~25-60 per request, that is
thousands of sequential round trips with no output and no progress — the
command is indistinguishable from a hang.

Default --limit to 100, matching `proxies replaced`. --limit 0 restores the
previous fetch-everything behavior, and the command now points at
`proxies download`, which renders a full list server-side in one request.

When the limit cuts the list short, say so on stderr so the cap is never
silent — stderr keeps the notice out of piped proxy lists.
The root command's examples redirect the proxy list to a file, which now
needs --limit 0 to stay a complete list.
CI pins staticcheck@2026.1 (v0.7.0) but installs go-version: stable, which
is now Go 1.27.1. staticcheck v0.7.0 cannot read Go 1.27's export data and
fails on stdlib packages before it reaches any repo code:

  internal error in importing "math/bits" (cannot decode "math/bits",
  export data version 4 is greater than maximum supported version 2)

This is unrelated to any code change — it reproduces on an unmodified main.
2026.2.1 supports the current toolchain and reports nothing on this repo.
@meznaric

Copy link
Copy Markdown
Contributor Author

Heads-up: this PR also unblocks CI

The first run here failed lint for a reason unrelated to the change:

internal error in importing "math/bits" (cannot decode "math/bits",
export data version 4 is greater than maximum supported version 2)

CI pins staticcheck@2026.1 (v0.7.0) but installs go-version: stable, which is now Go 1.27.1. staticcheck v0.7.0 cannot read Go 1.27's export data, so it dies on stdlib packages before reaching any repo code.

Verified pre-existing: running that exact staticcheck version against an unmodified origin/main in a clean worktree reproduces the same five errors. main has been red since Go stable moved past 1.26 — the last green run on it was 2026-07-28.

Bumped the pin to 2026.2.1, which supports the current toolchain and reports nothing on this repo (checked on both main and this branch).

Worth considering separately: the pinned linter against a floating stable Go will rot again the next time Go bumps its export format.

@meznaric
meznaric merged commit 7a409ee into main Sep 14, 2026
4 checks passed
@meznaric
meznaric deleted the limit-proxies-list-by-default branch September 14, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant