TestProjectListFilterByOrgAndProject is intermittent: depends on project list ordering

[leggetter]

Summary

TestProjectListFilterByOrgAndProject (test/acceptance/project_use_test.go:302, tag project_use) fails intermittently. It failed once in a full slice-0 run, then passed both in isolation and in a full slice-0 re-run of the identical commit.

Evidence

Against the merged fix/rc-exercise-findings at 238c157:

Run	Result
Full slice 0 (12 tags, 391-test suite)	FAIL TestProjectListFilterByOrgAndProject
Same test alone, -tags project_use -run TestProjectListFilterByOrgAndProject	PASS (1.52s)
Full slice 0, re-run, same commit	PASS (541s)

Slices 1 and 2 passed both times. So this is not a code regression — it is the test.

I do not have the failing assertion text. My run filtered output to --- FAIL lines, so the assertion message was discarded. Anyone reproducing should capture full -v output.

Why it looks fragile

The test makes two separate project list calls and assumes the result is stable between them:

1. Call one returns the full list; it takes fullList[0] — whichever project happens to be first.
2. It derives filter substrings from the first character of that project's org and project name.
3. Call two applies those substrings, and every returned row is asserted to contain them.

Two things could break that without any code being wrong:

• Ordering is not guaranteed. Nothing pins the order of project list, so "the first project" can differ between calls or between runs.
• Single-character substrings match broadly. The assertion loops over every returned row, so a one-character filter that matches many projects makes the test progressively more sensitive to the account's project set — which grows as test projects accumulate.

The CLI key is account-wide, so this test sees every project in every org the test user belongs to. That set is shared state, and other work in the same suite can change it.

Suggested direction

Make the test self-contained rather than dependent on whatever project list returns first: filter on a substring derived from a project the test can identify deterministically, or assert only that the filtered set is a subset of the full set matching the substring, rather than asserting over rows whose membership depends on ordering.

Why it matters

The acceptance suite is the release gate. A test that fails roughly one run in two erodes that gate: it trains people to re-run rather than investigate, which is how a real failure gets waved through.

This is the second intermittent acceptance test observed — see #387 for the other.

Related

• TestTelemetryGatewayIssueGetProxy fails intermittently and its failure output cannot be diagnosed #387 — TestTelemetryGatewayIssueGetProxy fails intermittently
• Acceptance tests leak sources and destinations: ~36k of each in the CI Gateway project #362 — acceptance tests leak resources, which grows the shared project set this test reads

---

Filed by Claude on Phil's behalf, from release-candidate testing.