Experiment Request: The Things That Count

[clesmian]

Hi everyone,

at FUZZING workshop in February we presented our registered report "The Things That Count: Coverage Evaluation Under the Microscope" (https://www.ndss-symposium.org/wp-content/uploads/fuzzing2026-30.pdf) where we try to see how accurate our coverage measurements are. To complete our proposed experiments and evaluation we'd like to include data from the FuzzBench cloud infrastructure.

Could you run an experiment using the fuzzers aflplusplus, honggfuzz, libafl, and libfuzzer? We'd be interested in a run with all 23 benchmarks for the standard duration of 24h. Reviewers requested us to run 30 trials, however, we'd be also happy with 10 trials in case 30 should exceed the available capacity.

Thanks in advance! Happy to discuss, in case there's any questions!

The changes we've made are mainly small fixes we've needed to make our local builds/runs succeed. We think they might also be useful to upstream for others.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[jonathanmetzman]

Hi, fuzzbench is broken now and I no longer work on it. There are no other maintainers so I don't think it will be possible to run your experiment. I'm sorry for this and I wish you luck in finding a good way to eval your work.

[addisoncrump]

Hey, we're starting to organise some resources so we can support groups through the support gap. Do you still need to run these? (see: https://github.com/AFLplusplus/fuzzbench and https://github.com/AFLplusplus/fuzzbench-results/tree/2026-06-09-aflpp-regress-mopt)

[clesmian]

Hey, thanks for the offer and organizing resources for the folks that need them! We were mainly interested to do an additional run on the cloud infrastructure to see whether there might be any differences between local runs within docker and the architecture used by the service.

As I assume you also use the docker based setup, we don't expect replicating the same setup on different hardware to provide any more meaningful insights. So we don't want to block your resources for anyone that can make better use of them.

That said, I will try to upstream any further fixes to https://github.com/AFLplusplus/fuzzbench that we make during our experiments.