Use the bugscache trigram index in search and cut N+1 queries in the job details panel#9689
Open
camd wants to merge 2 commits into
Open
Use the bugscache trigram index in search and cut N+1 queries in the job details panel#9689camd wants to merge 2 commits into
camd wants to merge 2 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9689 +/- ##
==========================================
+ Coverage 82.66% 82.68% +0.02%
==========================================
Files 626 626
Lines 36753 36793 +40
Branches 3348 3333 -15
==========================================
+ Hits 30383 30424 +41
+ Misses 5988 5987 -1
Partials 382 382 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Bugscache.search() filtered with __icontains, which Django compiles to UPPER(summary) LIKE UPPER(%s) on PostgreSQL. Because the predicate is on UPPER(summary) rather than the bare column, the planner could not use the gin_trgm_ops index on bugscache.summary and ran a full sequential scan on every call (once per distinct error term in the bug suggestions panel). Add a TrigramILike lookup that applies ILIKE directly to the column, with the same wildcard escaping as __icontains, so the existing trigram GIN index serves the search. On the prod replica (52k rows) this turns a ~125ms sequential scan into a ~5ms bitmap index scan for typical terms. Behaviour is unchanged: the existing search() parametrized tests (literal _/%% handling, case-insensitivity) still pass.
Two follow-on query fixes in the job details panel path: - Bugscache.serialize() ran a jobmap.count() per internal-issue row, so serializing a search result fired one query per internal issue. Add Bugscache.internal_occurrences() to compute all counts in a single grouped query and pass them into serialize(). - JobNoteSerializer.failure_classification_id used a SlugRelatedField on the related FailureClassification, loading the whole related row per note just to read its id (which is already the FK column). Switch to PrimaryKeyRelatedField (pk-only, no query). For the push_notes detail view, which reads failure_classification.name, add the classification to select_related so the name is joined instead of fetched per note.
camd
force-pushed
the
camd/query-optimization-failure-lines
branch
from
July 18, 2026 17:20
36ba1f3 to
85c36e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimizes the database queries behind the job details / failure summary panel, where the
bug_suggestionsendpoint has been reported as slow. The main fix makes an already-shipped index actually get used; two smaller commits remove N+1 queries on the same path.1.
Bugscache.search()never used the trigram index (main fix)search()filtered withsummary__icontains, which Django compiles toUPPER(summary) LIKE UPPER(%s)on PostgreSQL. Because the predicate is onUPPER(summary)rather than the bare column, the planner cannot use thegin_trgm_opsindex onbugscache.summary(added in #9555) and runs a full sequential scan on every call — andsearch()runs once per distinct error term in the bug suggestions panel.A small
TrigramILikelookup appliesILIKEdirectly to the column (same wildcard-escaping as__icontains), so the existing trigram GIN index serves the query. No migration required.Measured on the prod replica (
bugscache≈ 52,800 rows / 106 MB)EXPLAIN (ANALYZE, BUFFERS)for three representative terms:test-verify(generic)shutdownleaks(leak)Shared buffers per call drop from ~5,724 to 76–702. The cost goes from O(table size) — which grows unbounded as bugs accumulate — to O(matches).
2. Batch the per-row occurrences count
Bugscache.serialize()ranjobmap.count()for every internal-issue row, i.e. one query per internal issue in a search result. Addedinternal_occurrences()to compute all counts in a single grouped query and pass them intoserialize().3. Remove N+1 in job note serialization
JobNoteSerializer.failure_classification_idusedSlugRelatedField(slug_field="id"), loading the whole relatedFailureClassificationper note just to read its id (already present as the FK column). Switched toPrimaryKeyRelatedField(pk-only, no query).push_notesreadsfailure_classification.name, so its queryset nowselect_relateds the classification instead of fetching it per note.Testing
Bugscache.search()parametrized tests pass, confirming behaviour parity (literal_/%handling and case-insensitivity are unchanged).ILIKEon the bare column (noUPPER(...)wrapping) and escapes wildcards.internal_occurrencesverifying per-bug counts, window exclusion, and that it runs in a single query.error_summary, note, and job-note suites pass.ruffclean.