Commit 9960dbd
committed
Add benchmarking scripts
useragents.txt sample file kindly provided by @DailyMats out of
DailyMotion's data (2023-04-26).
The provided scripts allow:
- Testing the cache hit rate of various cache configuration (algorithm
and size) on sample files, this script uses a dummy parser and is
thus extremely fast.
- Benchmarking the average entry processing of various parser
configurations (base parser + cache algoritm + cache size) on sample
files, this is a much slower script but provides a a realistic
evaluation, and allows using custom rules (`regexes.yaml` files) to
check their impact on the performance of a given base parser.
Also added a script for testing threaded parsing, as expected this
gets 0 gain over the normal stuff because of the GIL (and re2
seemingly doesn't release the GIL either, though I don't know how
beneficial it would be at ~30us per call).
May be more useful with 3.13, or possibly with a regex-based extension
releasing the GIL, at least the basis for testing things out will be
here.1 parent e719a7e commit 9960dbd
5 files changed
Lines changed: 75591 additions & 1 deletion
0 commit comments