Skip to content

Benchmark parsing PHPDocs with Tally - #344

Merged
ondrejmirtes merged 2 commits into
2.4.xfrom
tally-benchmarks
Nov 14, 2026
Merged

ondrejmirtes merged 2 commits into
2.4.xfrom
tally-benchmarks

Conversation

@ondrejmirtes

Copy link
Copy Markdown
Member

Adds Tally benchmarks that guard the performance fixes of the last months. Tally counts instructions, cycles, and memory under CodSpeed's CPU simulation instead of timing them. The numbers barely move between CI runs, so the job fails on a regression of a few percent, which timing noise would hide.

Benchmark Input What it guards
LexerBench::benchTokenizeTypicalPhpDocs every tenth PHPDoc of the dev dependencies (491 docs) 148ceff with e59b0c8: 47.39M → 33.42M instructions, memory halved
SymfonyConfigReferenceBench::benchParse the config/reference.php Symfony 7.4 generated for an example app, parsed with line attributes like PHPStan does 1427af4: 977M → 482M instructions, then 3fcdcf0: → 199M

Numbers from one machine, measured at the commit before each fix and at the fix. For the Symfony file, parsing took 2,040 ms before 1427af4, 730 ms after it, and 132 ms after 3fcdcf0. I replayed both benchmarks on every tag since, from 2.3.5 to 2.4.1, and on the current 2.4.x: nothing has regressed since the fixes.

The setup:

  • Tally needs PHP 8.2, so it lives in its own Composer project in tools/tally, like phplrt in tools/phplrt. make bench-install installs it and make bench runs it. tally.php loads the library's autoloader.
  • The Benchmarks workflow runs on ubuntu-24.04 with PHP 8.5, because the counts depend on the image and the PHP build. It copies the pinned actions, hardening step, triggers, and concurrency of build.yml. actionlint 1.7.12 and zizmor report nothing.
  • benchmarks/ and tally.php are export-ignored.
  • The benchmarks have no expected numbers yet: the first run of the job prints a patch that adds the numbers it measured. I'll add a commit with them, so later runs compare against CI's own numbers.

Tally has no tagged release yet, so tools/tally requires dev-release/1.x, which includes the builds of CodSpeed's Valgrind.

Tally counts instructions, cycles, and memory under CodSpeed's CPU
simulation, so the numbers barely move between CI runs and a regression of
a few percent fails the build instead of disappearing in timing noise.

The two benchmarks guard the recent performance fixes:

- LexerBench lexes every tenth PHPDoc of the dev dependencies: 148ceff
  (with e59b0c8) took it from 47.39M to 33.42M instructions.
- SymfonyConfigReferenceBench parses the config/reference.php that
  Symfony 7.4 generates, one large array shape full of generics, as PHPStan
  does with line attributes: 1427af4 took it from 977M to 482M
  instructions, 3fcdcf0 to 199M.

Tally needs PHP 8.2, so it lives in tools/tally like phplrt in
tools/phplrt, and tally.php loads the library's autoloader. The Benchmarks
workflow runs it on a fixed image and PHP version, because the counts
depend on both. The expected numbers come from its first run: it prints a
patch that adds them.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d9be-190b-7065-bc04-21b38804941c
Taken from the tally-results artifact of the workflow's first run, so later
runs compare with numbers of the same image and PHP build.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d9be-190b-7065-bc04-21b38804941c
@ondrejmirtes
ondrejmirtes merged commit a0d180e into 2.4.x Nov 14, 2026
66 of 67 checks passed
@ondrejmirtes
ondrejmirtes deleted the tally-benchmarks branch November 14, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant