Chat Image Search

[Vjay15]

Schema should include

• Image Embeddings
• Caption (Indexed)
• Caption Embeddings
• Message ID/Message Link
• Timestamp
• pk

TODO

• Update Docker compose.yml file to add ollama and SmolVLM
• Enable PGVector, pg-trgm(for captions)
• Embedding size to be matching the SmolVLM model embed size.
• Embed the captions and image using ollama with SmolVLM.
• For a text search query generate embeddings and find similarity with Caption(use trigram), Caption and Image Embeddings.
• For a image search query generate embeddings and find similarity with Caption and image embeddings.

[kashifulhaque]

• Deduplication: Store the hash of each image too (use a perpetual hashing algo for this, it can handle cropped, resized and compressed images well
• Use temporal filter when searching:
  • "images from last month" (the timestamp column would work for this)
  • "images shared by username" (can add a username column too, tightens the search)
  • "find xyz image from this channel"
  • "find images similar to this" (takes an image as input and finds similar)

Potential table schema (feel free to modify)

- message_link
- image_url
- perceptual_hash
- caption
- caption_embeddings
- image_embeddings
- timestamp
- username
- channel_id
- server_id

[kashifulhaque]

vlms might not work nicely with ollama, we should prob go with https://huggingface.co/openai/clip-vit-base-patch32

EDIT: During my tests, I saw that most libraries are downloading pytorch (that's a huge dependency to download and a space hog). Fastembed seems like the way to go 🫠