Skip to content

feat(search): semantic image search with CLIP embeddings - #3316

Draft
dschmidt wants to merge 1 commit into
mainfrom
feat/semantic-image-search
Draft

feat(search): semantic image search with CLIP embeddings#3316
dschmidt wants to merge 1 commit into
mainfrom
feat/semantic-image-search

Conversation

@dschmidt

@dschmidt dschmidt commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Semantic image search: semantic:"dog on a beach" finds images by content, no tags needed, any language the model covers.

Indexing: a ClipExtractor decorator (around basic/tika) embeds images via a CLIP inference service (immich machine-learning API). Querying: the clause text is embedded into the same space and images are ranked by cosine similarity, bleve via faiss KNN behind the new vectors build tag, OpenSearch via knn_vector. The clause is split off the parsed KQL tree and composes with the rest of the query: the filter part scopes the neighbor search and owns the totals, hybrid rankings fuse via RRF, purely semantic queries return just the ranking. Vector size is index schema (512, startup probe hard-fails on mismatch); in bleve the raw vector lives in a stored-only sibling field so it survives move/delete/restore.

Enable with SEARCH_EXTRACTOR_CLIP_URL; bleve needs a build with ENABLE_VECTORS=true (faiss stage in Dockerfile.multiarch); existing content needs a force reindex.

Open: configurable dims/model, originals vs thumbnails to the service, capability flag for clients, web search bar wraps input as name:"*...*", OR/NOT position degrades to AND, faiss release/CI story, immich-ml is AGPL (external unmodified service).

@codacy-production

codacy-production Bot commented Aug 16, 2026

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 1 critical

Alerts:
⚠ 1 issue (≤ 0 issues of at least minor severity)

Results:
1 new issue

Category Results
Security 1 critical

View in Codacy

🟢 Metrics 148 complexity

Metric Results
Complexity 148

View in Codacy

🟢 Coverage 42.28% diff coverage · +0.18% coverage variation

Metric Results
Coverage variation +0.18% coverage variation (-1.00%)
Diff coverage 42.28% diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (993265b) 88137 20641 23.42%
Head commit (4993c9c) 88669 (+532) 20929 (+288) 23.60% (+0.18%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#3316) 570 241 42.28%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@dschmidt
dschmidt changed the base branch from tmp/refactor-search-mapping to refactor/search-mapping August 18, 2026 17:30
@dschmidt
dschmidt force-pushed the feat/semantic-image-search branch from 72257a7 to 4afdbe3 Compare August 18, 2026 17:30
@dschmidt
dschmidt force-pushed the refactor/search-mapping branch from a7ed42f to e5c12a9 Compare August 19, 2026 12:55
@dschmidt
dschmidt force-pushed the refactor/search-mapping branch 5 times, most recently from e4724ac to 9503884 Compare August 31, 2026 11:40
Base automatically changed from refactor/search-mapping to main August 31, 2026 13:12
@dschmidt
dschmidt force-pushed the feat/semantic-image-search branch from 4afdbe3 to b5699ac Compare September 3, 2026 00:51
semantic:"..." in KQL embeds the query text (immich-ml, multilingual CLIP)
and ranks image vectors by cosine similarity: bleve via faiss KNN behind the
new vectors build tag (RRF fusion, vector round-trip through a stored-only
field), OpenSearch via knn_vector plus client-side RRF. The filter part of
the query keeps its meaning and stays the only source of totals and facets.
@dschmidt
dschmidt force-pushed the feat/semantic-image-search branch from b5699ac to 4993c9c Compare September 3, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant