feat: PG full-text search columns (PPT-2644, 1/2) - #324
Merged
Conversation
…earch (PPT-2644) Adds a generated, stored tsvector column + GIN index to every table whose rest-api index route is searchable, replacing the Elasticsearch write path. 'simple' text-search config on both sides (no stemming — entity names, identifiers, emails; matches the guests.tsv_search precedent). Secrets and encrypted content are deliberately excluded from every vector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Elasticsearch's analyzer segmented uri/file-path fields, so their parts were individually searchable — PG 'simple' keeps '/path' as one lexeme. Split on punctuation (keeping the raw value) for mod.uri, driver.file_name/default_uri, repo.uri, shortener.uri and signage_plugin.uri. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t migration master's 9ff188d renumbered the AI-support migration to 20260806100500000, the exact version this migration was using. Micrate versions must be unique, and the search-vector migration must sort after everything on master, so it moves to 20260810100500000. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
|
LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(2026-08-10: rebased onto current master — zero conflicts — and the migration renumbered
20260806100500000→20260810100500000; master's #323 moved the AI-support migration onto our old number. The newai_*tables deliberately get no search vectors: they back no rest-api search route.)Part 1 of PPT-2644 (migrate search from Elasticsearch to PostgreSQL FTS). Additive and safe to merge first — nothing reads the new columns until the rest-api PR lands.
What
One migration (
20260810100500000_add_search_vector_columns.sql):search_vector tsvectorcolumn + GIN index on the 25 tables whose rest-api index routes are searchable (trigdeliberately excluded — trigger instances have no searchable text of their own; their search goes through the parent trigger via a query-time join)placeos_fts_join(text[])(arrays → text;array_to_stringis only STABLE so generated columns can't call it directly) andplaceos_fts_email(text)(guests-style[@._]tokenization + the raw address, so bothreevesand the whole address match)DownsectionDesign notes (details in the plan):
'simple'config on both write and query side — entity names/identifiers/emails must not be stemmed; matches the existingguests.tsv_searchprecedent (PPT-54 / feat: PPT-54 Add full text search capabilities to guest model #278)id::textis in every vector (Backoffice sendsidin every fields list)settings_string. Several of those are searchable via ES today; that ends here on purpose.setweight— the read side orders by name, not rank, so weights would be dead codeA follow-up models major will remove
include Neuroplastic, thees_type:/es_subfield:/es_ignore:attribute tags and theparent:tag once rest-api is off ES (part 2).⚠ Rollout note — prepared statements
Adding columns invalidates the result shape of
SELECT *prepared statements held by already-connected services: until their connections cycle, those queries fail with PG'scached plan must not change result type(observed live on a full local stack: rest-api/auth 500s after the ALTER, gone on service restart). Roll/restart the Crystal services after this migration applies (a normal deploy restart is sufficient) rather than leaving them on old connections. This applies to any wide migration, but 25 altered tables makes it near-certain here.Testing
spec/search_vector_spec.cr(5 examples): prefix matching on names/descriptions, array columns (features/tags/keys), email tokenization + whole-address lexeme, secrets-not-indexed negative, and the modules↔driver EXISTS join pattern rest-api will useplaylist_spec.cr:170(suite-order dependent; 13/0 standalone; fresh-DB A/B 4/4 green both ways) anduser_spec.cr:137(concurrent admin-delete advisory-lock timeout; 1-in-3 failure rate reproduced identically on a detached origin/master worktree under load)Full migration plan + research (11 reports): https://gist.github.com/camreeves/4b3cb2cffd2544e894ec34dcdae799a6