Phase 4: Advanced Retrieval Optimization for RAG Pipeline

[deoxyforge]

This PR improves context retrieval quality for intent extraction and test generation.

Changes:

• Increased Top-K retrieval for richer context
• Removed noisy and low-quality chunks
• Prioritized functions and classes for better relevance
• Eliminated duplicate chunks
• Added debug logging with chunk previews

Impact:

• Better context → improved intent detection → smarter test generation
• Reduced noise and redundancy in retrieved results

Builds on Phase 3 improvements.

[github-actions]

Automated Code Review Comment:

Bugs

• None

Improvements

• Safer collection creation using get_or_create_collection instead of create_collection
• Added safety check for empty documents in query_embeddings
• Improved filtering and cleaning of retrieved documents
• Removed duplicates from retrieved documents

Suggestions

• Consider adding error handling for chromadb client and collection operations
• The MIN_CHUNK_LENGTH threshold could be made configurable
• The query_embeddings function could be further optimized by using more advanced filtering or ranking techniques
• The debug logs could be made more informative or configurable for different environments

Code looks good ✅

---

Suggested Tests

import pytest
from your_module import store_embeddings, query_embeddings

def test_store_embeddings():
chunks = ["chunk1", "chunk2"]
embeddings = [[1, 2], [3, 4]]
store_embeddings(chunks, embeddings)

def test_query_embeddings():
query_embedding = [1, 2]
results = query_embeddings(query_embedding)
assert isinstance(results, list)

def test_query_embeddings_empty():
collection = client.get_or_create_collection(name="empty")
query_embedding = [1, 2]
results = collection.query(query_embeddings=[query_embedding], n_results=5)
assert results.get("documents", []) == [[]]

def test_query_embeddings_min_length():
query_embedding = [1, 2]
chunks = ["a" * 29, "b" * 31]
embeddings = [[1, 2], [3, 4]]
store_embeddings(chunks, embeddings)
results = query_embeddings(query_embedding)
assert len(results) == 1

def test_query_embeddings_k():
query_embedding = [1, 2]
chunks = ["a" * 31, "b" * 31, "c" * 31]
embeddings = [[1, 2], [3, 4], [5, 6]]
store_embeddings(chunks, embeddings)
results = query_embeddings(query_embedding, k=2)
assert len(results) == 2

[haddybhaiya]

changes looks good , but merge will happen into a distinct branch -closing!!