cookbooks: Falcon AI, Agent Command Center, and Evaluation notebooks - #6
Open
SuhaniNagpal7 wants to merge 17 commits into
Open
cookbooks: Falcon AI, Agent Command Center, and Evaluation notebooks#6SuhaniNagpal7 wants to merge 17 commits into
SuhaniNagpal7 wants to merge 17 commits into
Conversation
- semantic-caching: add Install intro, remove em/en dashes - evaluation/eval-correction-loop: document full create_custom_evals response shape - drop fallback-and-weighted-routing notebook from this PR (kept locally for later)
URL slug rename to match the renamed product concept (eval dataset -> golden dataset). Notebook renamed via git mv (history preserved). Internal href references in the other Falcon AI notebooks updated to the new slug.
…-loop Notebook prose matches the revamped MDX: - Outcome-first intro paragraph - 'What is an eval correction loop?' section - Per-step explainable framing (the why before the how) - Subtler sample data so the demo actually lands (is_helpful passes the bad rows on surface form; custom eval catches them; agreement 50% -> 100%) - Full create_custom_evals response shape in the print comment - Em/en dashes removed throughout
…ching Notebook prose matches the revamped MDX: - Outcome-first intro paragraph - 'What is Agent Command Center?' section with L1/L2 explanation - Per-step explainable framing (the why before the how) - Dashboard navigation corrected to 'Gateway -> Providers -> Cache' - L1 Backend (memory) vs L1 layer (always exact-match) distinction - Em/en dashes removed throughout
… API field names
End-to-end programmatic version of the docs cookbook:
- Step 1: judge model comparison across turing_flash / turing_small / turing_large
on a representative span so you can pick the right tier with evidence
- Step 2: list projects + list eval configs via the explicit
list_custom_eval_configs action, with safe_json + unwrap helpers that handle
the {status, result} FAGI wrapper
- Step 3: POST /tracer/eval-task/ with the verified serializer field names
(project, evals, run_type, sampling_rate, spans_limit, filters)
- Step 4: POST /tracer/user-alerts/ with verified field names
(metric_type, metric, project, threshold_type, threshold_operator,
critical_threshold_value, warning_threshold_value, alert_frequency)
- All API responses unwrapped through the {status, result} convention
- safe_json helper prints status + body snippet on non-JSON responses
- Em/en dashes removed throughout
…rified eval inputs
- Three security evals scoring three sample attacks (prompt-injection,
DAN-jailbreak, social-engineering)
- Verified required-input shapes per template:
prompt_injection={input}, answer_refusal={input, output},
is_harmful_advice={output}
- score_attack helper bundles all three evals per row
- Failure-filter loop surfaces only the rows where the agent broke through
- Closing prompt-patch diff shows the security rules to add to the system
prompt and re-run the suite against
- Em/en dashes removed throughout
…ith verified agent-opt + ai-evaluation API
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds six executable Colab notebooks matching the new cookbook docs pages.
New notebooks:
falcon-ai/end-to-end.ipynb- traced agent + dashboard lifecycle walkthroughfalcon-ai/context-aware-debugging.ipynb- traced agent that exposes a hallucination failurefalcon-ai/eval-datasets-from-traces.ipynb- traced classification agent for dataset curationcommand-center/fallback-and-weighted-routing.ipynb- OpenAI SDK +agentcc.GatewayConfigfor weighted load balancing and provider failovercommand-center/semantic-caching.ipynb- exact and semantic caching via gateway response headersevaluation/eval-correction-loop.ipynb- baseline eval, find disagreements, register a calibrated custom eval, re-score and measure agreementAll notebooks have been run end-to-end on Python 3.11 with the live
gateway.futureagi.comand theai-evaluationSDK. Outputs included in committed files where they demonstrate the workflow.Test plan
falcon-ai/end-to-end.ipynbend-to-end with a freshFI_API_KEYevaluation/eval-correction-loop.ipynband confirm baseline-vs-calibrated agreement deltas🤖 Generated with Claude Code