Supporting sharing vector embeddings in IIIF manifests / IIIF Images

[glenrobson]

In the AI4LAM conference Aaron Cope (SFO Museum) made a case for supporting Vector Embeddings in Manifests or IIIF Images as a way to allow cross collection searching using embeddings.

There is also this proposal from Northwestern which hasn't been shared with the wider community as far as I'm aware for a possible extension to IIIF to support embeddings as an annotation:

https://github.com/nulib-labs/iiif-embeddings-extension?tab=readme-ov-file

Brendan mentions the following use cases:

• Finding Images Similar to a Drawn Region: A user draws a rough shape on an image and searches for other images containing similar visual content based on their embeddings.
• Searching for Documents Based on Thematic Similarity: A researcher wants to find historical maps that discuss similar topics or depict similar geographical features, leveraging embeddings generated from associated text or visual content.
• Recommending Related Manuscript Pages: When viewing a page of a digitized manuscript, the system suggests other pages with semantically similar content based on embeddings of the text or illuminations.

[tpendragon]

This is interesting. I think we'd have to find a really clear way to reference the same model though - maybe like a combined checksum of the files relevant to running it? The same model with different quantizations or tensor format may result in different results and even be incompatible.

[azaroth42]

I agree with Trey: It's interesting but there's minimal interoperability because institutions will have different requirements for the models they use, and the embeddings for different models aren't comparable, and thus don't actually allow cross-collection searching.

Seems like an extension or seeAlso to prove the value would be the next step. We can't mandate an embedding model (it would instantly be out of date) and we can't compare (at least easily, and certainly not at run-time) across embedding models.