[3.x] Speed up initializing the lexer

[andreybolonin]

Lexer::getOperatorRegex() currently calls array_merge() for every expression parser, copying the tokens collected so far on every iteration. Collect the per-parser token arrays first and merge them once, avoiding repeated copying of the accumulated list.

The existing merge semantics, token ordering, duplicate handling, longest-operator-first sorting, and regex construction are preserved. Add coverage for overlapping operator names, shared aliases, and built-in operators alongside custom operators.

Performance

Measured against 428a16179, with PHP 8.5.11 CLI on macOS arm64, CLI OPcache disabled and JIT disabled. The default environment has 48 expression parsers, 49 operator tokens, 46 unique operator tokens. Both versions run in the same PHP process. Each result is the median of 9 batches, alternating before/after execution order, after 20 warm-up calls per version. Times below are microseconds per tokenize call, not batch totals.

The workload repeats this 118-byte template 0, 1, 10, 100, or 1,000 times:

<li>{{ item.name|upper }}: {% if item.price >= 10 and item.active %}{{ item.price * 2 }}{% else %}n/a{% endif %}</li>

First tokenization of a new lexer, reusing an already initialized environment:

Source size	Tokens, including EOF	Calls per version per batch	Before, µs/call	After, µs/call	Change in elapsed time
0 B	1	5,000	23.895	21.012	−12.07%
118 B	37	3,000	47.166	43.950	−6.82%
1,180 B	352	500	241.058	237.908	−1.31%
11,800 B	3,502	60	2,155.726	2,158.142	+0.11%
118,000 B	35,002	8	21,325.667	21,393.656	+0.32%

The saving is approximately 3 µs per lexer initialization. For the empty source, the nine batch averages ranged from 23.740–25.281 µs before and 20.748–21.359 µs after. For the 118-byte source, they ranged from 46.883–48.055 µs before and 43.876–44.883 µs after. On larger sources, tokenization dominates and the difference is within run-to-run variation; this is an initialization optimization, not a throughput improvement proportional to template size.

Repeated tokenization with an already initialized lexer (same batch sizes):

Source size	Before, µs/call	After, µs/call	Change in elapsed time
0 B	0.468	0.463	−1.10%
118 B	22.254	22.338	+0.38%
1,180 B	214.902	215.316	+0.19%
11,800 B	2,127.324	2,127.935	+0.03%
118,000 B	21,558.240	21,584.011	+0.12%

There is no meaningful improvement on this path because operator regex initialization happens once per lexer instance. A separate test including a new environment + a new lexer + first tokenization of the 118-byte source measured 110.267 → 107.215 µs/call (−2.77%), with 2,000 calls per batch. These local measurements do not claim an equivalent improvement in full rendering or cached-template execution.

Validation

• Full suite: vendor/bin/simple-phpunit — 2,872 tests, 7,017 assertions, 4 skipped, no failures (existing legacy deprecation notices reported).
• vendor/bin/phpstan analyse --no-progress — no errors.
• vendor/bin/php-cs-fixer fix --config=.php-cs-fixer.dist.php --dry-run --diff src/Lexer.php tests/LexerTest.php — no changes needed.
• Differential tokenization of 665 template sections from the existing fixtures: identical serialized token streams, or identical exception class/message, between the base lexer and this branch.

Reproduce the benchmark and differential fixture check

Run from the repository root after installing Composer dependencies. Save the PHP script below as /tmp/twig-lexer-benchmark.php, then run:

git show 428a16179:src/Lexer.php > /tmp/twig-lexer-before.php
php -d opcache.enable_cli=0 /tmp/twig-lexer-benchmark.php

The base lexer is renamed in memory so both versions can run in the same process. OPcache is explicitly disabled for this comparison. The script also prints the minimum/maximum batch averages and checks the fixture corpus.

<?php
require getcwd().'/vendor/autoload.php';
// Save the base version first: git show 428a16179:src/Lexer.php > /tmp/twig-lexer-before.php
$baseline = file_get_contents('/tmp/twig-lexer-before.php');
eval(substr(str_replace('class Lexer', 'class BaselineLexer', $baseline), 5));
use Twig\Environment;
use Twig\Loader\ArrayLoader;
use Twig\Source;
function measure(callable $callback, int $iterations): float {
    $start = hrtime(true);
    for ($i = 0; $i < $iterations; ++$i) { $callback(); }
    return (hrtime(true) - $start) / $iterations / 1000;
}
function compare(string $label, callable $before, callable $after, int $iterations): void {
    for ($i = 0; $i < 20; ++$i) { $before(); $after(); }
    $samples = [[], []];
    for ($round = 0; $round < 9; ++$round) {
        foreach ($round % 2 ? [1, 0] : [0, 1] as $side) {
            $samples[$side][] = measure([$before, $after][$side], $iterations);
        }
    }
    sort($samples[0]); sort($samples[1]);
    $a = $samples[0][4]; $b = $samples[1][4];
    printf("%s | n=%d | %.3f -> %.3f us/op | %+.2f%% time | range %.3f..%.3f / %.3f..%.3f\n", $label, $iterations, $a, $b, 100*($b/$a-1), $samples[0][0], $samples[0][8], $samples[1][0], $samples[1][8]);
}
$env = new Environment(new ArrayLoader());
$parsers = iterator_to_array($env->getExpressionParsers());
$tokens = [];
foreach ($parsers as $parser) {
    $tokens = array_merge($tokens, Twig\ExpressionParser\ExpressionParsers::getOperatorTokensFor($parser));
}
printf("PHP %s; %d parsers; %d tokens (%d unique); CLI OPcache=%s; JIT=%s\n", PHP_VERSION, count($parsers), count($tokens), count(array_unique($tokens)), ini_get('opcache.enable_cli'), ini_get('opcache.jit'));
$unit = "<li>{{ item.name|upper }}: {% if item.price >= 10 and item.active %}{{ item.price * 2 }}{% else %}n/a{% endif %}</li>\n";
foreach ([0, 1, 10, 100, 1000] as $repeats) {
    $source = new Source(str_repeat($unit, $repeats), 'bench');
    $old = new Twig\BaselineLexer($env); $new = new Twig\Lexer($env);
    $stream = $old->tokenize($source);
    $count = 1;
    while (!$stream->isEOF()) { $stream->next(); ++$count; }
    if (serialize($old->tokenize($source)) !== serialize($new->tokenize($source))) { throw new RuntimeException('Token mismatch'); }
    $label = strlen($source->getCode())." bytes, $count tokens";
    $n = [0=>5000, 1=>3000, 10=>500, 100=>60, 1000=>8][$repeats];
    compare("first tokenize, $label", fn()=>(new Twig\BaselineLexer($env))->tokenize($source), fn()=>(new Twig\Lexer($env))->tokenize($source), $n);
    compare("warm tokenize, $label", fn()=>$old->tokenize($source), fn()=>$new->tokenize($source), $n);
}
$source = new Source($unit, 'bench');
compare('new Environment + first tokenize, 1 unit', fn()=>(new Twig\BaselineLexer(new Environment(new ArrayLoader())))->tokenize($source), fn()=>(new Twig\Lexer(new Environment(new ArrayLoader())))->tokenize($source), 2000);
// Differential coverage over the existing integration/lexer fixture corpus.
$checked = 0;
foreach (new RecursiveIteratorIterator(new RecursiveDirectoryIterator(getcwd().'/tests/Fixtures', FilesystemIterator::SKIP_DOTS)) as $file) {
    if (!$file->isFile()) { continue; }
    $contents = file_get_contents($file->getPathname());
    preg_match_all('/--TEMPLATE(?:\([^\n]*\))?--\r?\n(.*?)(?=\n--[A-Z]|\z)/s', $contents, $matches);
    foreach ($matches[1] as $template) {
        $source = new Source($template, $file->getPathname());
        $results=[];
        foreach ([Twig\BaselineLexer::class,Twig\Lexer::class] as $class) {
            try { $results[]=serialize((new $class($env))->tokenize($source)); }
            catch (Throwable $e) { $results[]=$e::class.':'.$e->getMessage(); }
        }
        if ($results[0] !== $results[1]) { throw new RuntimeException('Fixture mismatch: '.$file->getPathname()); }
        ++$checked;
    }
}
echo "Differential fixture templates checked: $checked\n";

Memory measurements

Compared 428a16179 with this MR's exact head in separate, fresh PHP processes (not two renamed classes loaded together). PHP 8.5.11 CLI on macOS arm64, JIT disabled, with CLI OPcache both off and on. Five independent processes per version/workload/OPcache mode, alternating execution order; values below are medians.

Inputs, the environment, and extension initialization are set up before the measured interval. memory_reset_peak_usage() resets the peak immediately before the operation. Reported used-heap peaks come from memory_get_peak_usage(false); reserved peaks come from memory_get_peak_usage(true). Absolute peaks include the existing baseline heap; “incremental peak” subtracts the heap immediately before the operation. Result objects/strings remain alive when memory is sampled. Output verification happens afterward, so its serialization buffers are excluded. Process max RSS includes startup/setup and is sampled before verification. OPcache shared usage is reported separately because it is not part of the PHP used-heap metric.

All before/after output hashes matched. These are memory measurements only; process startup is excluded from the earlier time benchmarks.

Full first tokenization, using the same 118-byte template repeated 0, 1, 10, 100, or 1,000 times as in the time benchmark. Every value in this table is identical before and after:

Source size, bytes	Used-heap peak, KiB, OPcache off	Used-heap peak, KiB, OPcache on	Reserved peak, MiB, off / on
0	2,257.852	1,013.234	4 / 2
118	2,331.688	1,023.070	4 / 2
1,180	2,412.836	1,104.219	4 / 2
11,800	3,196.070	1,887.453	4 / 4
118,000	11,702.414	10,393.797	12 / 12

Isolated getOperatorRegex(), invoked once via reflection on a new lexer with expression parsers already initialized:

Additional parsers (2 tokens each)	OPcache	Incremental peak before, bytes	Incremental peak after, bytes	Difference
0	off	8,064	15,016	+6,952 B
0	on	8,064	14,760	+6,696 B
100	off	50,496	53,256	+2,760 B
100	on	50,496	53,000	+2,504 B
1,000	off	446,208	446,208	+0 B
1,000	on	446,208	446,208	+0 B

With the default 48 parsers, batching the token arrays increases this isolated temporary peak by 6,952 B without OPcache / 6,696 B with OPcache (about 6.8 / 6.5 KiB). The extra arrays are transient: retained heap after this method is 896 B in both versions. The larger peak later in full tokenization is unchanged. Full empty-source tokenization with 100 and 1,000 additional parsers also had identical used-heap and reserved peaks before/after.

After a warm-up followed by 100 tokenizations of an 11,800-byte source on the same lexer, retained heap relative to the warm-up baseline is 96 B in both versions (the final token stream is held); incremental peak is 128 B in both versions. This workload shows no additional accumulation introduced by the change.

OPcache-reported shared used memory is 64 B higher in the after-version processes. Peak reserved PHP memory is unchanged in every tested workload.

Process max RSS for the largest full-operation workload (includes startup/setup; five fresh processes per side):

OPcache	Before median, MiB (range)	After median, MiB (range)
off	36.625 (36.562–36.750)	36.609 (36.547–36.625)
on	40.000 (39.984–40.062)	40.000 (39.969–40.016)

RSS has page-level/process-startup variation and should not be used to attribute byte-sized differences to this patch; the allocator/retained-heap figures above show the allocation differences directly.

Reproduce the memory benchmark

Install Composer dependencies, save the PHP and Python scripts below as /tmp/twig-memory-probe.php and /tmp/twig-memory-run.py, then run from the repository root:

git show 428a16179:src/Lexer.php > /tmp/twig-memory-lexer-before.php
git show 8df9ea9e2:src/Lexer.php > /tmp/twig-memory-lexer-after.php
git show 428a16179:src/Compiler.php > /tmp/twig-memory-compiler-before.php
git show 8ba033841:src/Compiler.php > /tmp/twig-memory-compiler-after.php
python3 /tmp/twig-memory-run.py

The runner launches 420 fresh PHP processes in total across both MRs. It loads the original class name from the selected snapshot before the Composer autoloader can load it, so each process contains exactly one implementation of the measured class. All other source files must match the MR base; the time/memory changes in these two MRs are confined to the selected Compiler/Lexer methods. Inputs and dependencies outside those classes are shared identically across each comparison. PHP 8.2+ is needed for memory_reset_peak_usage(); the reported results use PHP 8.5.11. The scripts save all individual measurements to /tmp/twig-memory-results.json and print per-case medians.

<?php
require getcwd().'/vendor/autoload.php';
[$script, $target, $side, $scenario, $count] = $argv;
$count = (int) $count;
require "/tmp/twig-memory-$target-$side.php";
$env = new Twig\Environment(new Twig\Loader\ArrayLoader());
// Keep fixture construction and extension initialization outside the measured interval.
$parsers = $env->getExpressionParsers();
if ($target === 'lexer') {
    if ($scenario === 'operators' || $scenario === 'extended') {
        $extra = [];
        for ($i = 0; $i < $count; ++$i) {
            $extra[] = new Twig\ExpressionParser\Prefix\UnaryOperatorExpressionParser(Twig\Node\Expression\Unary\NotUnary::class, 'extra_'.$i, 50, aliases: ['alias_'.$i]);
        }
        $parsers->add($extra);
        unset($extra);
    }
    $unit = "<li>{{ item.name|upper }}: {% if item.price >= 10 and item.active %}{{ item.price * 2 }}{% else %}n/a{% endif %}</li>\n";
    $source = new Twig\Source(str_repeat($unit, $scenario === 'tokenize' ? $count : ($scenario === 'repeat' ? 100 : 0)), 'bench');
    $lexer = new Twig\Lexer($env);
    $method = new ReflectionMethod(Twig\Lexer::class, 'getOperatorRegex');
    $operation = $scenario === 'operators' ? fn() => $method->invoke($lexer) : fn() => $lexer->tokenize($source);
    if ($scenario === 'repeat') {
        $lexer->tokenize($source);
    }
} else {
    $compiler = new Twig\Compiler($env);
    $env->setCompiler($compiler);
    $unit = '{% set x = [1, 2, 3, 4, 5] %}{{ x[0] }}' . "\n";
    $template = $scenario === 'compile' ? ($count ? str_repeat($unit, $count) : 'Hello, world!') : str_repeat($unit, 100);
    $source = new Twig\Source($template, 'bench');
    $env->getLoader()->setTemplate('bench', $template);
    if ($scenario === 'repr') {
        $value = $count ? range(0, $count - 1) : 123;
        $operation = fn() => $compiler->reset()->repr($value)->getSource();
    } else {
        $operation = fn() => $env->compileSource($source);
        if ($scenario === 'repeat') {
            $operation();
        }
    }
}
gc_collect_cycles();
$usedBefore = memory_get_usage(false);
$reservedBefore = memory_get_usage(true);
memory_reset_peak_usage();
$iterations = $scenario === 'repeat' ? 100 : 1;
for ($i = 0; $i < $iterations; ++$i) {
    unset($result);
    $result = $operation();
}
// Capture before hashing/serializing output so verification buffers do not affect the peak.
$usedAfter = memory_get_usage(false);
$peakUsed = memory_get_peak_usage(false);
$reservedAfter = memory_get_usage(true);
$peakReserved = memory_get_peak_usage(true);
gc_collect_cycles();
$usedAfterGc = memory_get_usage(false);
// Capture RSS before output verification allocates serialization buffers.
$rss = getrusage()['ru_maxrss'];
if (PHP_OS_FAMILY === 'Linux') {
    $rss *= 1024;
}
$opcacheStatus = opcache_get_status(false);
$opcacheUsed = $opcacheStatus ? $opcacheStatus['memory_usage']['used_memory'] : null;
if (is_string($result)) {
    $digest = hash('sha256', $result);
} else {
    $digest = hash('sha256', serialize($result));
}
echo json_encode([
    'target'=>$target, 'side'=>$side, 'scenario'=>$scenario, 'count'=>$count,
    'opcache'=>(int) ini_get('opcache.enable_cli'),
    'used_before'=>$usedBefore, 'used_after'=>$usedAfter, 'used_after_gc'=>$usedAfterGc,
    'peak_used'=>$peakUsed, 'peak_delta'=>$peakUsed-$usedBefore,
    'retained_delta'=>$usedAfterGc-$usedBefore,
    'reserved_before'=>$reservedBefore, 'reserved_after'=>$reservedAfter, 'peak_reserved'=>$peakReserved,
    'max_rss'=>$rss, 'opcache_used'=>$opcacheUsed, 'digest'=>$digest,
]), "\n";

import json, subprocess, statistics, pathlib
cases = [('lexer','operators',n) for n in [0,100,1000]]
cases += [('lexer','tokenize',n) for n in [0,1,10,100,1000]]
cases += [('lexer','extended',n) for n in [100,1000]]
cases += [('lexer','repeat',0)]
cases += [('compiler','repr',n) for n in [0,100,10000,100000]]
cases += [('compiler','compile',n) for n in [0,1,10,100,1000]]
cases += [('compiler','repeat',0)]
all_results=[]
for opcache in [0,1]:
    for target,scenario,count in cases:
        pair={}
        for repeat in range(5):
            for side in (['before','after'] if repeat%2==0 else ['after','before']):
                cmd=['php',f'-d',f'opcache.enable_cli={opcache}','-d','opcache.file_update_protection=0','-d','opcache.jit=disable','/tmp/twig-memory-probe.php',target,side,scenario,str(count)]
                result=json.loads(subprocess.check_output(cmd,text=True))
                all_results.append(result)
                pair.setdefault(side,[]).append(result)
        if {v['digest'] for side in pair.values() for v in side}.__len__()!=1:
            raise RuntimeError(f'Output mismatch: {target} {scenario} {count} {opcache}')
        def m(side,key):return statistics.median(v[key] for v in pair[side])
        print(f'{target} {scenario} {count}, OPcache={opcache}: peak used {m("before","peak_used"):.0f} -> {m("after","peak_used"):.0f}, delta {m("after","peak_used")-m("before","peak_used"):+.0f}; reserved {m("before","peak_reserved"):.0f} -> {m("after","peak_reserved"):.0f}; RSS {m("before","max_rss"):.0f} -> {m("after","max_rss"):.0f}',flush=True)
        pathlib.Path('/tmp/twig-memory-results.json').write_text(json.dumps(all_results,indent=2))