Bound the rate-limit retry budget at 5 minutes - #149
Open
MichaelGHSeg wants to merge 6 commits into
Open
MichaelGHSeg wants to merge 6 commits into
MichaelGHSeg wants to merge 6 commits into
Conversation
The 12 hour default was a backstop on the assumption a retry count would stop us reaching it. Across the SDKs rate-limited attempts are deliberately uncounted, so a duration is what actually bounds that path — C# was the only one with a count at all, and it sat at 100. Five minutes matches the counted path's ~4 minute worst case. MaxRetryInterval drops to 60s. At 300s it equalled the whole budget, so one sleep consumed it and the rate-limit path gave a single attempt. The wait is clamped to the end of the episode's budget, since ShouldUploadBatch checks elapsed time before waiting. One test assertion is deliberately inverted rather than adjusted: RateLimitCountIsReachedLongBeforeTheDurationBudget asserted that the count trips first and the duration is unreachable. That was the right invariant when the duration was 12 hours; now the duration is the operative limit and the count is the backstop, so it asserts the reverse and says why. 264 unit tests and the 82-test e2e suite pass. The e2e suite failed 6 tests on an earlier run under heavy machine load and passed clean on a re-run at normal speed — it is timing-sensitive, which is worth knowing for CI.
Three problems. The notes described changes between states that never shipped, so a customer read that a default moved from 12 hours to 5 minutes when only the 5 minutes was ever released. They referred to other SDKs, which means nothing to someone reading one library's notes. And they had accumulated over several passes into contradictions — Retry-After was documented as capped at both 300s and 60s, and the rate-limit budget as both 12 hours and 5 minutes. Rewritten to describe the behaviour this version has, in a consistent structure: upgrade notes that need action first, then retry handling, then everything else. Entries covering fixes to code that has not shipped are dropped, since there is nothing for a reader to compare against.
RateLimitStartTime was persisted as an absolute timestamp and restored on load. That was survivable at a 12 hour budget and is not at 5 minutes: an app closed for longer than the budget now loads an already-expired episode and discards the batch on its first flush, having never retried it while actually running. The clock measures how long this process has spent retrying, and time while the process was not running is not that. It is now in-memory only — neither written nor read. The retry counts still persist, so a batch cannot be retried indefinitely across restarts; a relaunch gets a fresh duration budget but inherits the counted one. This only affects targets that persist state at all. Server-side use generally swaps the disk store for an in-memory one, where the field was already per-process. Also corrects a comment I wrote in the previous commit. It described the duration as a last-ditch guard reached long after the count — true at 12 hours, and the inverse of what the same commit made true. The duration is now what stops retrying and the count is the backstop, which is what the test one file over asserts. 264 unit tests and the 82-test e2e suite pass.
The clamp in HandleRateLimitResponse had no coverage at all. Deleting it outright left all 264 tests passing: the two test files this change touched check config validation and arithmetic on default constants, and neither calls the state machine. The one part of the change with any logic in it was the one part unguarded. Two tests now drive HandleResponse directly through a fake clock. The first opens an episode, advances to one second before the budget ends, sends a 429 asking for sixty, and asserts the wait lands on the episode deadline — it fails with the clamp removed. The second asserts a wait that comfortably fits is passed through untouched, so the clamp cannot degenerate into truncating every Retry-After to the deadline. 266 unit tests and the 82-test e2e suite pass.
Applying the team convention to my own work from today. The comments explaining these changes had accumulated into potted histories: why a value had been twelve hours, what a test used to assert, which path used to be unreachable. Six months from now none of that resolves to anything — the diff and the commit messages hold it, and the comment should say why the code is the way it is. What stayed is what a maintainer would undo without it: that Kernel#sleep raises on a negative interval, that Thread#wakeup only interrupts a sleep already in progress, that OkHttp's reads are governed by SO_TIMEOUT so an interrupt does not reach them, and that inverting one assertion would make the duration budget unreachable again. Comments only, no behaviour change.
Capping at 60s meant waiting less than the server asked for, which does not make the next attempt more likely to succeed — it just sends more requests at something already rate-limiting us. Against a Retry-After of 180s inside a 5 minute budget it turns 3 requests into 6; against 300s it turns 2 into 6. The cap is a guard against an absurd header, not a second budget. How long we keep trying is max_rate_limit_duration's job, and the clamp to the remaining budget already stops a single wait running past it, so the cap now rarely binds at all. It also bought nothing for the client this was partly aimed at: with no background thread, a shorter cap turns one long wait into several short ones for the same total blocking time and more requests. Tests that pinned 60 are updated, and each SDK gains one asserting that a Retry-After inside the cap is used as given rather than shortened.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Retry handling has two budgets. Responses carrying
Retry-Aftertake a rate-limit path bounded byMaxRateLimitDurationand a retry count; everything else takes a counted exponential-backoff path.MaxRateLimitDurationwas 12 hours, intended as a last-ditch guard behind the count. Rate-limited attempts are uncounted elsewhere in the fleet, and a duration is what genuinely bounds that path, so 12 hours against a 100-count meant the count was doing all the work and the guard was unreachable.What
RateLimitConfig.MaxRateLimitDurationdefaults to 5 minutes, in line with the counted path's ~4 minute worst case. The duration is now the operative limit and the count is the backstop behind it —MaxRateLimitDurationIsTheOperativeLimitNotTheCountasserts that relationship rather than the two numbers.MaxRetryIntervaldrops to 60s. At 300s it equalled the whole budget, so one wait consumed it.ShouldUploadBatchtests elapsed time before the wait.RateLimitStartTimeis no longer persisted. It measures how long this process has spent retrying, and time while the process was not running is not that. Carrying it over was survivable at a 12 hour budget; at 5 minutes an app closed for six minutes loads an already-expired episode and discards the batch on its first flush, having never attempted it while running. The retry counts still persist, so a batch cannot be retried indefinitely across restarts. This only affects targets that persist at all — server-side use generally swaps in an in-memory store.Testing
266 unit tests and the full 82-test shared e2e suite pass.
The clamp previously had no coverage: deleting it outright left all 264 tests green, because the tests this change touched check config validation and arithmetic on defaults rather than the state machine. Two tests now drive
HandleResponsethrough a fake clock — one asserts the wait lands on the episode deadline and fails with the clamp removed, the other asserts a wait that fits is passed through untouched so the clamp cannot degenerate into truncating everyRetry-After.CI cannot currently run the e2e suite — the private
sdk-e2e-testscheckout lost its token during the CI-hardening work — so it was run locally. That suite is timing-sensitive; it produced spurious failures on a machine under heavy load and passed cleanly at normal speed.Notes
Release notes describe the behaviour this version has rather than the delta from an unreleased state, since the retry feature ships in this same version. The exception is retries being enabled by default, which is a real change from 2.6.0 and carries its own note.
<Version>still needs bumping off the published 2.6.0 before tagging.