Repository navigation
Benchmark parsing PHPDocs with Tally - #344
Merged
Merged
Conversation
Tally counts instructions, cycles, and memory under CodSpeed's CPU simulation, so the numbers barely move between CI runs and a regression of a few percent fails the build instead of disappearing in timing noise. The two benchmarks guard the recent performance fixes: - LexerBench lexes every tenth PHPDoc of the dev dependencies: 148ceff (with e59b0c8) took it from 47.39M to 33.42M instructions. - SymfonyConfigReferenceBench parses the config/reference.php that Symfony 7.4 generates, one large array shape full of generics, as PHPStan does with line attributes: 1427af4 took it from 977M to 482M instructions, 3fcdcf0 to 199M. Tally needs PHP 8.2, so it lives in tools/tally like phplrt in tools/phplrt, and tally.php loads the library's autoloader. The Benchmarks workflow runs it on a fixed image and PHP version, because the counts depend on both. The expected numbers come from its first run: it prints a patch that adds them. Amp-Thread-ID: https://ampcode.com/threads/T-01a1d9be-190b-7065-bc04-21b38804941c
Taken from the tally-results artifact of the workflow's first run, so later runs compare with numbers of the same image and PHP build. Amp-Thread-ID: https://ampcode.com/threads/T-01a1d9be-190b-7065-bc04-21b38804941c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Tally benchmarks that guard the performance fixes of the last months. Tally counts instructions, cycles, and memory under CodSpeed's CPU simulation instead of timing them. The numbers barely move between CI runs, so the job fails on a regression of a few percent, which timing noise would hide.
LexerBench::benchTokenizeTypicalPhpDocsSymfonyConfigReferenceBench::benchParseconfig/reference.phpSymfony 7.4 generated for an example app, parsed with line attributes like PHPStan doesNumbers from one machine, measured at the commit before each fix and at the fix. For the Symfony file, parsing took 2,040 ms before 1427af4, 730 ms after it, and 132 ms after 3fcdcf0. I replayed both benchmarks on every tag since, from 2.3.5 to 2.4.1, and on the current
2.4.x: nothing has regressed since the fixes.The setup:
tools/tally, like phplrt intools/phplrt.make bench-installinstalls it andmake benchruns it.tally.phploads the library's autoloader.ubuntu-24.04with PHP 8.5, because the counts depend on the image and the PHP build. It copies the pinned actions, hardening step, triggers, and concurrency ofbuild.yml. actionlint 1.7.12 and zizmor report nothing.benchmarks/andtally.phpare export-ignored.Tally has no tagged release yet, so
tools/tallyrequiresdev-release/1.x, which includes the builds of CodSpeed's Valgrind.