Skip to content

Harden SQLite with CI-backed SQLancer regressions#435

Open
adamziel wants to merge 37 commits into
WordPress:trunkfrom
adamziel:codex/sqlancer-sqlite-fuzz
Open

Harden SQLite with CI-backed SQLancer regressions#435
adamziel wants to merge 37 commits into
WordPress:trunkfrom
adamziel:codex/sqlancer-sqlite-fuzz

Conversation

@adamziel

@adamziel adamziel commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Big picture

This PR turns SQLancer into a repeatable compatibility loop for SQLite Database Integration.

SQLancer generates MySQL statements, we keep the statements MySQL accepts, replay them through the SQLite driver, and turn SQLite-only failures into permanent regression coverage. The point is not to keep fuzzing on a local workstation. The branch moves that load into GitHub Actions and gives us a durable queue for anything the workflow finds next.

What changed

  • Added bin/run-sqlancer-sqlite-fuzz.sh, a bounded runner that starts disposable MySQL, captures SQLancer output, filters out MySQL-rejected statements, and replays the accepted prefix through the SQLite driver.
  • Added tests/fuzz/replay-sqlancer-log.php and tests/fuzz/append-sqlancer-finding.php so generated logs can be replayed and failures can be formatted as GitHub issue comments.
  • Added .github/workflows/sqlancer-sqlite-fuzz.yml, which runs on this fuzz branch, on a six-hour schedule, and by manual dispatch. It rotates SQLancer oracles/seeds and records newly found replay failures in one GitHub issue named SQLancer SQLite replay findings.
  • The findings issue is deduplicated by hidden failure hash and capped at 1000 finding comments, so the queue stays useful without growing without bound.
  • Added tests/e2e/specs/sqlancer-fuzz-regressions.test.js so reduced SQLancer failures also run through the real WordPress/plugin path, not only the package driver.
  • Fixed a set of driver compatibility gaps found by SQLancer around MySQL modifiers, numeric coercion, functional indexes, boolean expressions, COUNT(DISTINCT ...), ALTER TABLE ... RENAME, text prefix indexes, signed integer clipping, and non-transactional table implicit defaults.

CI fuzz loop

The expected workflow now is:

  1. GitHub Actions runs SQLancer on push and every six hours.
  2. If replay succeeds, the run stays green.
  3. If replay finds a SQLite-only failure, the workflow creates or updates the SQLancer SQLite replay findings issue with one comment for that failure.
  4. A human reduces the finding, fixes the behavior, and moves the reduced case into the package and/or WordPress e2e regression tests in this same PR.

Manual runs are still available when we want a specific oracle or seed:

gh workflow run sqlancer-sqlite-fuzz.yml \
  --ref codex/sqlancer-sqlite-fuzz \
  -f oracle=PQS \
  -f seed=20261207 \
  -f num_queries=1500 \
  -f max_generated_databases=1 \
  -f append_findings=true

How to test this branch

For local development, prefer targeted checks instead of running SQLancer:

cd packages/mysql-on-sqlite
./vendor/bin/phpunit tests/WP_SQLite_Driver_Tests.php --filter 'testSqlancer'

Useful static checks:

php -l packages/mysql-on-sqlite/src/sqlite/class-wp-pdo-mysql-on-sqlite.php
php -l packages/mysql-on-sqlite/tests/WP_SQLite_Driver_Tests.php
php -l tests/fuzz/append-sqlancer-finding.php
node --check tests/e2e/specs/sqlancer-fuzz-regressions.test.js
bash -n bin/run-sqlancer-sqlite-fuzz.sh
git diff --check -- . ':!.cao-tasks'

Validation so far

  • Package SQLancer regression slice: OK (25 tests, 198 assertions).
  • Focused MEMORY implicit-default regression: OK (2 tests, 12 assertions) with the adjacent HEAP case.
  • PHPCS passed for the touched driver, test, and SQLancer helper files.
  • node --check passed for the SQLancer e2e spec.
  • The SQLancer Actions workflow has already run successfully on this branch, including manual dispatch and branch-push runs.
  • The issue-backed findings update passed local YAML parsing, workflow run: block bash -n, helper PHP lint, PHPCS, and a synthetic failure-formatting smoke test.

Known limitations

  • The CI job records new SQLancer findings; it does not automatically reduce, fix, or commit regression tests for them.
  • Scheduled fuzzing is bounded by GitHub Actions timeouts. It is continuous coverage over time, not an infinite single process.
  • The findings issue is capped at 1000 entries. When it is full, the workflow stops appending new findings until humans reduce or clear old ones.
  • SQLancer can still generate failures that are too large to review directly. Those should stay in the findings issue until reduced into a readable test case.
  • The WordPress e2e file intentionally contains reduced representative cases, not every generated query from every SQLancer run.

@adamziel adamziel changed the title Harden SQLite with SQLancer fuzz regressions Harden SQLite with CI-backed SQLancer regressions Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant