test: drop wall-clock search-engine benchmark tests - #267
Merged
Conversation
testCommandSearchBenchmarkBeatsLegacyPipeline and testSwitcherSearchBenchmarkBeatsLegacyPipeline only asserted XCTAssertLessThan on elapsed milliseconds against a hand-rolled reference scan — no result-correctness assertions, and the optimized-vs-reference result equivalence they exercised is already covered by testOptimizedSearchMatchesReferencePipeline in the same file. Together they were 36.2s (27%) of measured unit-test body time and a CI-load flakiness risk (1.25x margin on shared runners). Removed the two tests plus their benchmark-only helpers (benchmarkElapsedMs, repeatedQueries), which were used by nothing else. Added a one-line pointer to the out-of-band performance harness (tests_v2/test_cpu_occlusion_throttle.py, docs/cpu-harness.md).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Removes two performance-benchmark tests from
CommandPaletteSearchEngineTests.swiftthat were burning 36.2 seconds — 27% of the entire unit suite's measured test-body time (132s total) — while asserting nothing about correctness. They compared elapsed milliseconds of the optimized search against a hand-rolled linear-scan "reference" pipeline with a1.25xmargin, which is also a CI-load flakiness risk on shared runners (timing assertions can trip under noisy neighbors).Summary
testCommandSearchBenchmarkBeatsLegacyPipelineandtestSwitcherSearchBenchmarkBeatsLegacyPipeline(were at lines 831/870).benchmarkElapsedMsandrepeatedQueries— used by nothing else in the file.testOptimizedSearchMatchesReferencePipelinealready asserts the optimized search matches the reference pipeline's results, for both the command and switcher corpora, across 9 queries — this is the coverage the benchmarks'_ = ...discarded-result calls never actually asserted on.tests_v2/test_cpu_occlusion_throttle.py/docs/cpu-harness.md, not in the unit suite.XCTSkip: a skipped test still compiles, and the audit found the dominant CI cost here isxcodebuildcompiling ~46K lines of test source (~9 of the job's ~10.5 minutes), not test execution — so this is a first cut at trimming volume, not a claim of large wall-time savings from these two tests alone.Test Plan
xcodebuild -project GhosttyTabs.xcodeproj -scheme programa-unit -configuration Debug -destination 'platform=macOS' test -only-testing:programaTests/CommandPaletteSearchEngineTests→ 24/24 tests pass (was 26 before), suite executes in 0.424s (down from 36.67s aggregate previously attributed to this suite).