From 64df0ae97fc56989dcb18e9bca3138806b297351 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Tue, 15 Sep 2026 10:14:33 +0100 Subject: [PATCH 1/3] ci: run acceptance tests only when CLI or test code changes The acceptance suite ran on every PR to main/next, so docs-only and repo-furniture PRs spent three API-key-backed matrix jobs (12m timeout each) plus the telemetry job to test a binary nobody had changed. Add an allowlist `paths:` filter covering everything the suite actually builds and asserts on: main.go, go.mod, go.sum, pkg/**, test/**, tools/** and .github/workflows/**. An allowlist rather than paths-ignore: a directory nobody thought about fails to trigger the suite visibly, instead of a denylist silently running it forever until someone adds an exclusion. Safe to skip because `unit-test` is the only required status check on main's ruleset, so a skipped acceptance run leaves no PR pending. workflow_dispatch deliberately keeps no filter -- it is the escape hatch for the case this filter creates (see #395). Closes #240 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BnrKWZQASV7bFJ4oGWwmo9 --- .github/workflows/test-acceptance.yml | 34 +++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/.github/workflows/test-acceptance.yml b/.github/workflows/test-acceptance.yml index 2866448c..8d32d2fc 100644 --- a/.github/workflows/test-acceptance.yml +++ b/.github/workflows/test-acceptance.yml @@ -5,8 +5,42 @@ on: branches: - next - main + # Only run when a change could actually affect what the suite tests. See #240. + # + # This is an allowlist, not a paths-ignore denylist, and deliberately so: a + # new top-level directory nobody thought about simply does not trigger the + # suite, and the omission is visible (the run is missing) rather than + # silently burning three API-key-backed slots on every docs PR. A denylist + # fails the other way -- it runs everything until someone remembers to add + # an exclusion. + # + # Skipping is safe here because `unit-test` (in test.yml, which has no path + # filters) is the only required status check on main's ruleset, so a PR that + # skips acceptance is not left permanently pending. + paths: + # The binary under test: acceptance helpers run `go build -o .` and + # `go run main.go`, so the entrypoint, every package and the module graph + # are what that build resolves. + - "main.go" + - "go.mod" + - "go.sum" + - "pkg/**" + # The tests and their helpers. test/ holds only test/acceptance. + - "test/**" + # The reference generator gates on `--help` text, so a change here can + # move output the tests assert on. + - "tools/**" + # A workflow change can break the run itself, and the acceptance run is a + # two-file chain (test-acceptance.yml -> acceptance.yml). Matching the + # whole directory means adding another reusable workflow to that chain + # does not require remembering to extend this list. + - ".github/workflows/**" # Allow running the acceptance suite against any branch by hand. # + # workflow_dispatch takes no paths filter, by design: it is the escape hatch + # for exactly the case the filter above creates, so it must fire regardless of + # what changed. + # # The pull_request trigger above does not fire when a PR's head branch is # updated by merging another PR into it, so a stacked release branch can end # up with no acceptance run at all. workflow_dispatch is the only way to From a59587ed9551b3c1f5626c992b78af16494e6f94 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Tue, 15 Sep 2026 10:27:14 +0100 Subject: [PATCH 2/3] ci: run acceptance on merge to main as well The suite has only ever triggered on pull_request -- zero runs in history with headBranch main -- so main's head is strictly a commit nothing ran acceptance against. The PR gate is inherited on merge, and that inheritance is only as current as the PR's last pull_request event. That is not hypothetical. Merging a PR into another PR's head branch fires no pull_request event, so #378 sat with 20 commits and 75 changed files against a stale run, and the code about to ship had never been through its own gate. test.yml already runs on push to main for exactly this reason, with the comment "so the merged result is tested, not just each PR branch in isolation. A PR based on stale main can merge into a combination that no PR run exercised." The argument was never extended to the stronger suite. Uses the same paths filter as the pull_request trigger, so a docs merge does not spend a full suite run -- which is what makes this affordable and is why it belongs in this PR rather than a separate one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BnrKWZQASV7bFJ4oGWwmo9 --- .github/workflows/test-acceptance.yml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/.github/workflows/test-acceptance.yml b/.github/workflows/test-acceptance.yml index 8d32d2fc..41ffda0f 100644 --- a/.github/workflows/test-acceptance.yml +++ b/.github/workflows/test-acceptance.yml @@ -35,6 +35,29 @@ on: # whole directory means adding another reusable workflow to that chain # does not require remembering to extend this list. - ".github/workflows/**" + # Run on merge to main so the merged result is tested, not just each PR + # branch in isolation -- the same reasoning test.yml already applies to the + # unit and build jobs. A PR based on stale main can merge into a combination + # no PR run exercised, and a merge commit is strictly a commit nothing ran + # acceptance against. + # + # This closes a real gap: merging a PR into another PR's head branch fires no + # pull_request event, so a release branch can reach main carrying changes its + # last acceptance run never saw. + # + # Same paths filter as above, so a docs merge does not spend a full suite run. + push: + branches: + - main + paths: + - "main.go" + - "go.mod" + - "go.sum" + - "pkg/**" + - "test/**" + - "tools/**" + - ".github/workflows/**" + # Allow running the acceptance suite against any branch by hand. # # workflow_dispatch takes no paths filter, by design: it is the escape hatch From 7955b390574c444f8e27f615b0d9776e7d418dd1 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Tue, 15 Sep 2026 11:33:30 +0100 Subject: [PATCH 3/3] ci: serialise acceptance runs so they stop colliding Every caller of this reusable workflow -- pull requests, merges to main, manual dispatch, and the release gate -- exercises the same three Hookdeck projects with the same three API keys. The API allows 240 requests/minute and a full slice makes thousands of calls, so two overlapping runs exhaust it and both degrade. It has happened twice in two days. Once as all four jobs timing out with zero assertion failures. Once as a single named test failing -- a config that came back empty -- while a release ran concurrently against the same projects; that one looks exactly like a regression and sends you hunting through the auth code for a bug that is not there. Both times the same commit passed on a re-run with nothing else in flight. A note in the acceptance README tells a human how to recognise this after the fact. It does not prevent it, and writing it down did not stop the second occurrence. cancel-in-progress is deliberately false: release.yml calls this as its publish gate, and cancelling that to make room for a pull request is the wrong trade. Queueing costs wall-clock and nothing else. Placed on the reusable workflow rather than each caller so no future caller can be added outside the group. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BnrKWZQASV7bFJ4oGWwmo9 --- .github/workflows/acceptance.yml | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/.github/workflows/acceptance.yml b/.github/workflows/acceptance.yml index 7779fb04..54bcd7ad 100644 --- a/.github/workflows/acceptance.yml +++ b/.github/workflows/acceptance.yml @@ -6,6 +6,28 @@ name: Acceptance Tests on: workflow_call: +# Serialise every acceptance run repo-wide. +# +# All callers -- pull requests, merges to main, manual dispatch, and the release +# gate -- exercise the SAME three Hookdeck projects using the same three API +# keys. The API allows 240 requests/minute and a full slice makes thousands of +# calls, so two runs overlapping exhaust the limit and both degrade. +# +# The failure is nasty because it does not look like rate limiting. It surfaces +# as ordinary assertion failures in whichever slice lost the race -- a config +# that came back empty, a filter that returned nothing -- sending whoever reads +# it hunting for a regression that does not exist. It has happened twice: once +# as four jobs timing out with zero assertion failures, and once as a single +# named test failing while a release ran concurrently. +# +# cancel-in-progress is deliberately false. Cancelling would be faster, but the +# release workflow calls this as its publish gate, and killing that run to make +# room for a pull request is the wrong trade. Queueing costs wall-clock and +# nothing else. +concurrency: + group: acceptance-suite + cancel-in-progress: false + jobs: acceptance: strategy: