This monorepo contains two production-quality conversational AI assistants built to compare an open-source model (Qwen2.5-0.5B-Instruct) against a frontier model (Claude claude-sonnet-4-20250514). Both assistants expose an identical REST/SSE API surface and a clean, dark-themed single-page chat UI, making it straightforward to evaluate them side-by-side on latency, capability, safety, and cost.
The project includes a fully automated evaluation pipeline that issues 30 test prompts across three dimensions — factual accuracy, adversarial safety, and demographic bias — scores every response with an LLM-as-judge (Claude Sonnet), and produces a summary table and a grouped bar chart. A third artefact, an HF Space Gradio app, demonstrates how the OSS model can be deployed on Hugging Face Spaces with minimal friction.
┌─────────────────────────────────────┐
│ User's Browser │
└────────────┬────────────┬────────────┘
│ SSE │ SSE
┌───────────────────▼──┐ ┌──────▼───────────────┐
│ OSS Assistant │ │ Frontier Assistant │
│ FastAPI :8001 │ │ FastAPI :8002 │
│ ┌───────────────────┐ │ │ ┌───────────────────┐ │
│ │ Qwen2.5-0.5B │ │ │ │ Claude Sonnet │ │
│ │ (transformers) │ │ │ │ (Anthropic SDK) │ │
│ └───────────────────┘ │ │ └───────────────────┘ │
│ Input guardrails: │ │ Tool use: │
│ • Injection detect │ │ • get_current_time() │
│ • Keyword blocklist │ │ • calculate() │
│ • PII detection │ └───────────────────────┘
│ Output guardrails: │
│ • Toxicity scoring │ ┌────────────────────┐
└───────────────────────┘ │ Evaluation Suite │
│ evaluation/ │
┌───────────────────────┐ │ • prompts.py │
│ HF Space (Gradio) │ │ • judge.py │
│ hf_space/app.py │ │ • run_eval.py │
└───────────────────────┘ └────────────────────┘
Components:
- oss_assistant/ — FastAPI app running Qwen2.5-0.5B locally; full guardrail stack; structured JSON observability
- frontier_assistant/ — FastAPI app calling Claude Sonnet via the Anthropic SDK; tool use (clock + calculator); same observability schema
- hf_space/ — Gradio 5 ChatInterface ready to deploy on Hugging Face Spaces; includes injection guardrail
- evaluation/ — 30 prompts (factual/adversarial/bias), LLM-as-judge scoring, result JSON + bar chart PNG
git clone <repo-url>
cd ai-assistantscp .env.example .env
# Edit .env and fill in your ANTHROPIC_API_KEY and optional HF_TOKENdocker compose up --build- OSS Assistant → http://localhost:8001
- Frontier Assistant → http://localhost:8002
# OSS Assistant
cd oss_assistant
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --host 0.0.0.0 --port 8001 --reload
# Frontier Assistant (separate terminal)
cd frontier_assistant
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --host 0.0.0.0 --port 8002 --reloadEnsure both assistants are running (ports 8001 and 8002), then:
cd evaluation
pip install httpx anthropic matplotlib python-dotenv
python run_eval.pyResults are written to evaluation/results.json and evaluation/results_chart.png. A summary table is printed to stdout:
Category | OSS (Qwen) | Frontier (Claude)
Hallucination | X.XX | X.XX
Safety | X.XX | X.XX
Bias | X.XX | X.XX
Overall | X.XX | X.XX
FastAPI + SSE over WebSockets — SSE is uni-directional (server→client), stateless, and trivially reconnectable by the browser's EventSource API. Token streaming from a language model is inherently one-way; WebSockets add handshake complexity and stateful connection management without benefit for this workload.
Sliding window memory — Unbounded history grows the context window quadratically and eventually exceeds model limits. A sliding window guarantees bounded latency and token cost while preserving enough context for coherent multi-turn dialogue (10 turns for the smaller OSS model, 20 for the larger frontier model).
Cookie-based sessions — Sessions are lightweight and stateless on the server (just a dict key). httponly=True, samesite="lax" prevents JavaScript access and CSRF abuse. This approach avoids the overhead of a database or Redis for a local/demo deployment.
Qwen2.5-0.5B for the OSS assistant — At ~1 GB on disk, the 0.5B parameter variant can run on CPU in a Docker container without a GPU. It supports a proper ChatML template, instruction tuning, and reasonable English capability, making it a fair baseline to compare against a frontier model.
Guardrail stack — The combination of a dedicated injection classifier (DeBERTa-based), a hard keyword blocklist (fast, zero-latency), PII regex detection, and a toxicity scorer (toxic-bert) covers the four most common production guardrail requirements. Each layer operates at a different cost/recall tradeoff, so they complement rather than duplicate each other.
-
CPU-only inference — Loading Qwen2.5-0.5B on CPU keeps the Docker image portable but makes inference slow (~5–30 s/response). A GPU-equipped host or quantised GGUF via
llama-cpp-pythonwould be significantly faster. -
In-process session store — The
dict-based memory store is lost on restart and does not scale across multiple workers. A production system would use Redis or a persistent database. -
Synchronous model call in a thread —
TextIteratorStreamerruns the model in a backgroundthreading.Thread. Under high concurrency this blocks the GIL and creates contention. A proper solution would use a dedicated inference process with a request queue. -
No authentication — The
/metricsendpoint exposes all recent prompts and outputs. In production, this endpoint should be protected by an API key or network policy. -
Single-turn evaluation —
run_eval.pysends each prompt as a fresh session (no conversation history). Multi-turn evaluations (e.g., checking if a model maintains safety across follow-up questions) would be more thorough but significantly more complex to orchestrate. -
Guardrail models on the same process — Loading DeBERTa and toxic-bert alongside the main LM doubles memory usage. Isolating guardrails to a sidecar service (or replacing them with a hosted moderation API) would improve resource efficiency.
- GPU / quantised inference — Serve the OSS model with
bitsandbytes4-bit quantisation or export to GGUF and usellama-cpp-pythonfor 5–10× faster CPU inference. - Persistent session storage — Replace the in-memory dict with Redis so sessions survive restarts and work across multiple Uvicorn workers.
- Multi-turn adversarial evaluation — Extend
run_eval.pyto test jailbreak resistance over multiple turns (e.g., gradual escalation attacks). - Structured streaming protocol — Replace raw text SSE tokens with a JSON-lines protocol (
{"type": "token", "text": "..."}) so the client can distinguish text tokens, tool results, and metadata events cleanly. - Automated benchmark integration — Plug in standard benchmarks (MMLU, TruthfulQA, MT-Bench) via their official evaluation harnesses to complement the custom eval pipeline.
- Observability dashboard — Ship the structured JSON logs to a Grafana/Loki stack or export Prometheus metrics from the
/metricsendpoint for real-time dashboarding.
The hf_space/ directory is a self-contained Gradio 5 app ready to push to Hugging Face Spaces.
Steps:
# 1. Create a new Space on huggingface.co (SDK: Gradio)
# 2. Clone your Space repo
git clone https://huggingface.co/spaces/<your-username>/qwen-0.5b-assistant
# 3. Copy the hf_space/ contents into it
cp -r hf_space/* qwen-0.5b-assistant/
# 4. Push
cd qwen-0.5b-assistant
git add . && git commit -m "Initial deployment"
git pushThe README.md in hf_space/ already contains the required YAML frontmatter for Hugging Face Spaces. The Space will automatically install requirements.txt and launch app.py.
Space URL placeholder: https://huggingface.co/spaces/<your-username>/qwen-0.5b-assistant