We should support [websearch](https://github.com/huggingface/chat-ui/blob/c6f34ec07d90ff68fc1aca543da0d08d1663970e/src/lib/server/tools/web/search.ts) and [fetch url](https://github.com/huggingface/chat-ui/blob/c6f34ec07d90ff68fc1aca543da0d08d1663970e/src/lib/server/tools/web/url.ts). Techniques: - [x] hybrid search (with simple concatenation of multiple retrievals in d3c5ca4cb30e543898d3269a5bd14b993f39d628 and reciprocal rank fusion in be2ee59b9eb12c4274f2cf995d9eadbae333e7a9) - [x] deduplication in d3c5ca4cb30e543898d3269a5bd14b993f39d628 - [ ] reranking - [ ] [query routing/rewriting/expansion](https://cdn-uploads.huggingface.co/production/uploads/6469c972a5dd10c9a49d683b/U3PKvTNnXtp63ZwlpmL8X.png) - [x] [small-to-big retrieval](https://www.llamaindex.ai/blog/towards-long-context-rag) (implictly supported) More advanced: - [ ] [Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) - [ ] [adaptive retrieval](https://supabase.com/blog/matryoshka-embeddings) - [ ] [embedding align](https://huggingface.co/blog/PLB/embedding-align-rag) - [ ] [Hypothetical Document Embeddings (HyDE)](https://arxiv.org/abs/2212.10496)
We should support websearch and fetch url.
Techniques:
More advanced: