feat(benchmarks): generate the README comparison table in CI - #535
Open
barjin wants to merge 5 commits into
Open
feat(benchmarks): generate the README comparison table in CI#535barjin wants to merge 5 commits into
barjin wants to merge 5 commits into
Conversation
…ng it The comparison table added in #523 was measured by hand, so it goes stale as soon as any of the compared projects ships a release and nobody can check it. This replaces the numbers with a harness that produces them: a local Node.js HTTP/2 origin, one benchmark script per ecosystem driving the latest published release of every client, and a script that rewrites the table between markers in the README. A monthly workflow runs the lot and opens a PR when the numbers move. Nothing is pinned and no lockfile is committed — the point is to compare what the ecosystems ship today, impit included. Two things the harness found that the hand-written table got wrong. got-scraping was reported as HTTP/1.1-only because HTTP/2 died with a GOAWAY after roughly a thousand requests; that was Node's Rapid-Reset mitigation on the test server counting got's per-response RST_STREAM, not a got-scraping defect, so it is switched off in the origin and got-scraping is now measured over h2 like everything else. And the origin reports its connection count, which shows cycletls opening a fresh TLS connection per request rather than reusing a warm one — previously invisible in its throughput number, now footnoted.
github-actions
Bot
added
the
t-tooling
…umbers The first CI run showed why best-of-N is the wrong statistic here: node-tls-client ranged from 900 to 3095 req/s across its eleven runs, so its best would have put it above impit while its median sits a third below. That spread is intrinsic to the client rather than machine noise, which is exactly the case best-of-N flatters. The table now quotes the median, the result files keep best and worst, and a client whose runs swing by more than 1.5x gets a footnote saying so. The committed numbers are now the ones the workflow measured on a runner instead of the ones from a loaded laptop.
barjin
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #469 — replaces the hand-measured table from #523 with a benchmark harness that measures the latest published clients against a local server in CI, monthly or on demand, and opens a PR when the numbers move.