Skip to content

fix(postgres): stop rebuilding every BM25 index on startup - #55

Merged
mudler merged 1 commit into
mudler:mainfrom
walcz-de:fix/bm25-text-config-compare
Sep 30, 2026
Merged

mudler merged 1 commit into
mudler:mainfrom
walcz-de:fix/bm25-text-config-compare

Conversation

@walcz-de

Copy link
Copy Markdown
Contributor

Problem

ensureBM25IndexConfig builds the expected string as text_config='<cfg>' (quoted) and looks for it in the output of pg_get_indexdef(). PostgreSQL renders the option unquoted:

BM25 index text_config differs, recreating index="idx_documents_pprls_docs_bm25" want="simple"
current_def="CREATE INDEX … USING bm25 (full_text) WITH (text_config=simple)"

The check can never match, so every startup drops and rebuilds every BM25 index, even when nothing changed. On our instance (~90 collections) that meant ~90 index rebuilds per start: 186 of these log lines over two restarts. It also hides the one case the check exists for, a real config change.

Fix

Extract the text_config value from the index definition with a small regex that accepts the quoted and the unquoted form (including schema-qualified configs such as public.de_en), and compare it case-insensitively. A different config still drops the index and triggers the rebuild, as before.

The bug came in with the env-configurable text_config (#52, ours).

Tests

rag/engine/postgres_bm25_config_test.go, pure function, no database needed:

  • unquoted form from pg_get_indexdef → match
  • quoted form → match
  • schema-qualified custom config → match
  • different config → no match (rebuild still happens)
  • no text_config at all → no match

go test ./rag/engine/ -args -ginkgo.focus=bm25IndexHasTextConfig: 5 passed. Without the fix the new tests do not build (the helper is new). gofmt, go vet clean.

🤖 Generated with Claude Code

ensureBM25IndexConfig compared the quoted form text_config='<cfg>'
against pg_get_indexdef(), which renders the option unquoted
(text_config=simple). The check could never match, so every startup
dropped and rebuilt every BM25 index and logged
'BM25 index text_config differs' with a current_def that already
carried the wanted config. On an instance with ~90 collections that is
~90 index rebuilds per start (186 such log lines over two restarts).

Extract the value from the index definition instead, accepting both the
quoted and the unquoted form, and compare it case-insensitively. A
different config still triggers the rebuild, as before.

The bug came in with the env-configurable text_config (mudler#52).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
@mudler
mudler merged commit 257b989 into mudler:main Sep 30, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants