Freshness-Aware RAG

Re-rank vector search results using time decay and source authority, not cosine similarity alone.

Most RAG pipelines return the chunks that look most like the question.
That fails for “what’s new?”, pricing changes, and anything where an old document can still be highly similar.

This library sits after your vector store and re-ranks candidates so recent, trusted sources can beat stale but similar ones.

score = (1 - w) × semantic_similarity
      + w × (freshness_score × exp(−age / half_life) × source_authority)

Pure vector search	With this re-ranker
Best match by embedding only	Best match and recency and trust
Old Medium post can rank #1 for “latest model”	Fresh Reuters / official docs preferred
All domains treated equal	Authority map (e.g. reuters.com > random blog)

---

Install

git clone https://github.com/pandeyvishwas51-oss/freshness-aware-rag.git
cd freshness-aware-rag
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python scripts/demo.py    # works offline, no database required
pytest tests/ -v

For live Weaviate + embeddings:

pip install -e ".[retrieval]"
docker compose up -d weaviate
python scripts/seed_demo_data.py

---

Usage

from src.retriever import FreshnessAwareRetriever

retriever = FreshnessAwareRetriever(
    weaviate_url="http://localhost:8080",
    embed_model="all-MiniLM-L6-v2",
    default_freshness_weight=0.3,  # 0 = pure semantic, 1 = pure freshness
)

chunks = retriever.retrieve(
    query="latest interest rate decision",
    top_k=10,
    freshness_weight=0.4,
    max_age_hours=72,
)

for c in chunks:
    print(c.url, "|", c.chunk_text[:100])

Wire into any LLM:

context = "\n\n".join(f"[{c.url}]\n{c.chunk_text}" for c in chunks)
prompt = f"Answer using only the sources below.\n\n{context}\n\nQuestion: {question}"
# → your model

No Weaviate? Rank in-memory items (tests, notebooks, custom stores):

scored = retriever.retrieve_from_items(items, top_k=5, freshness_weight=0.35)

HTTP:

python -m src.api.main
curl -X POST http://localhost:8001/v1/retrieve \
  -H "Content-Type: application/json" \
  -d '{"query":"latest OpenAI model","top_k":5,"freshness_weight":0.35}'

Step-by-step: docs/HOW_TO_USE.md

---

When to use this

Good fit	Weak fit
News / “what just happened” bots	Evergreen science Q&A (use low w)
Live web indexes for RAG	Static FAQ that never changes
Finance / markets assistants	Pure code search by symbol
Product docs & changelogs that move
Multi-source corpuses with uneven quality

Tuning: news-like queries → w ≈ 0.4 to 0.6. Long-lived docs → w ≈ 0.1 to 0.2.

---

How it works

                query
                  │
                  ▼
            embed query
                  │
                  ▼
     vector DB recall  (top k×3 candidates)
                  │
                  ▼
     re-rank each candidate:
       · semantic similarity
       · age → exponential time decay
       · domain → source authority
       · optional freshness_score from crawler
                  │
                  ▼
            top-k chunks → LLM

Core pieces:

• FreshnessAwareRetriever: search + re-rank
• scoring.py: pure math (easy to unit test)
• authority.py: domain trust / update frequency maps
• Weaviate schema helpers + optional FastAPI service

---

Project layout

src/retriever/     ranking engine
src/ingestion/     Weaviate schema & batch insert
src/api/           FastAPI service
src/shared/        RetrievedChunk models
scripts/demo.py    offline comparison demo
tests/             22 unit tests

---

FAQ

Does this replace my vector database?
No. It re-ranks what the DB already returns (or ranks a list you pass in).

Do I need a GPU?
Not for the default MiniLM embedding path; CPU is fine.

Is this an LLM?
No. Retrieval only, plug OpenAI, local models, LangChain, LlamaIndex, etc. on top.

What if I only care about similarity?
Set freshness_weight=0.

---

License

MIT