Skip to content

cookbook(tracing): add Chroma RAG evaluation notebook - #9

Open
SuhaniNagpal7 wants to merge 1 commit into
mainfrom
cookbook/evaluate-chroma-rag
Open

cookbook(tracing): add Chroma RAG evaluation notebook#9
SuhaniNagpal7 wants to merge 1 commit into
mainfrom
cookbook/evaluate-chroma-rag

Conversation

@SuhaniNagpal7

Copy link
Copy Markdown
Contributor

Summary

Companion notebook for the docs cookbook "Trace and Evaluate a Chroma RAG Pipeline with Future AGI" (future-agi/docs#TBD). Mirrors the docs walkthrough as a runnable `.ipynb`:

  1. Auto-instrument ChromaDB and OpenAI with `traceAI`.
  2. Build an in-memory Chroma collection with five short policy snippets.
  3. Run a retrieve-then-generate flow on `gpt-4o-mini`.
  4. Score the result with five Future AGI evaluators (`context_relevance`, `chunk_attribution`, `chunk_utilization`, `completeness`, `factual_accuracy`).

Includes interpretation, troubleshooting, and next-steps sections so the notebook stands on its own when opened from a Colab badge.

Branch convention

Following the pattern used by other docs-companion notebooks (e.g. `cookbook/quickstart-notebooks`), this notebook lives on its dedicated cookbook branch. The docs Colab badge will point at `/blob/cookbook/evaluate-chroma-rag/tracing/evaluate-chroma-rag.ipynb`.

Verification

The exact Python in this notebook was run end-to-end against a live Future AGI workspace before being committed. `register()` connected, `ChromaDBInstrumentor` + `OpenAIInstrumentor` instrumented, `agent` answered, and five eval scores landed (Context Relevance 1.0, Chunk Attribution 1.0, Chunk Utilization 0.6, Completeness 1.0, Factual Accuracy 1.0). Three spans landed in the FAGI dashboard.

Test plan

  • Open the Colab badge from the docs PR once both PRs are merged.
  • Run all cells top to bottom with `FI_API_KEY`, `FI_SECRET_KEY`, and `OPENAI_API_KEY` set; confirm one Chroma trace and one OpenAI trace land in the dashboard, plus the five-row score printout.

Companion notebook for the docs cookbook "Trace and Evaluate a Chroma
RAG Pipeline with Future AGI". Builds a small in-memory Chroma RAG
loop, auto-instruments ChromaDB and OpenAI with traceAI, and scores
the run with five Future AGI evaluators that diagnose retrieval and
generation independently.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant