Documentation | Quickstart | PyPI | Issues
SMOLTRACE is a benchmarking and evaluation framework for Smolagents, Hugging Face's lightweight agent library. It runs ToolCallingAgent and CodeAgent against curated or custom task datasets, auto-instruments everything with OpenTelemetry, and publishes results, traces, metrics, and a leaderboard — with zero configuration.
pip install smoltrace
export HF_TOKEN=hf_your_token
export OPENAI_API_KEY=sk-your_key # or MISTRAL_API_KEY, ANTHROPIC_API_KEY, ...
smoltrace-eval --model openai/gpt-4.1-nano --provider litellm --agent-type both --enable-otelThat's it. One command evaluates both agent types and automatically creates 4 datasets on the Hugging Face Hub:
{username}/smoltrace-results-{timestamp}— per-test-case outcomes{username}/smoltrace-traces-{timestamp}— OpenTelemetry spans{username}/smoltrace-metrics-{timestamp}— GPU / CO2 / cost time-series{username}/smoltrace-leaderboard— aggregate stats, updated every run
No repo names to configure — everything is derived from your HF_TOKEN.
- Zero configuration — only
HF_TOKENrequired; dataset names are auto-generated from your username - Multi-provider — LiteLLM (OpenAI, Anthropic, Mistral, Groq, ...), HF Inference API, local Transformers models on GPU, and Ollama
- OTEL native — auto-instrumented via genai-otel-instrument: spans, token counts, cost tracking, CO2 emissions
- GPU metrics — utilization, memory, temperature, and power tracked by default for local models, flattened for dashboards
- 20+ agent tools — web search, webpage reading, file system, text processing (
grep/sed/sort), and process/system tools (ps/curl/ping), with working-directory sandboxing - 3 benchmark datasets — quick tasks (13), comprehensive benchmark (132: GAIA/Math/SimpleQA), and an ops/SRE benchmark (24)
- Flexible output — push to HF Hub, save local JSON, or export to OpenSearch with typed mappings
- Parallel execution —
--parallel-workers 8for 10-50x faster API-model evaluations - Dataset lifecycle —
smoltrace-copy-datasetsto bootstrap,smoltrace-cleanupto safely prune old runs - HF Jobs ready — run on Hugging Face cloud hardware (CPU to A100) with pay-as-you-go billing
pip install smoltrace # core
pip install smoltrace[gpu] # + GPU metrics for local models
pip install smoltrace[opensearch] # + OpenSearch exportRequires Python 3.10+. For development installs and full requirements, see the installation guide.
# API model via LiteLLM (default provider)
smoltrace-eval --model openai/gpt-4 --provider litellm --agent-type tool
# Local GPU model via Transformers (GPU metrics on by default)
smoltrace-eval --model meta-llama/Llama-3.1-8B --provider transformers --agent-type both
# Local model via Ollama
smoltrace-eval --model qwen2.5-coder:3b --provider ollama --agent-type tool --enable-otel
# HF Inference API
smoltrace-eval --model meta-llama/Llama-3.1-70B-Instruct --provider inference --agent-type both
# Enable agent tools, filter by difficulty, run in parallel
smoltrace-eval \
--model openai/gpt-4.1-nano \
--provider litellm \
--enable-tools visit_webpage read_file grep \
--working-directory ./workspace \
--difficulty easy \
--parallel-workers 8 \
--enable-otelOther useful flags: --dataset-name (custom task dataset), --model-args temperature=0.7 seed=42, --prompt-yml (custom prompts), --mcp-server-url (MCP tools), --private (private datasets), --output-format hub|json|opensearch.
Full flag reference: CLI documentation.
import os
from smoltrace.core import run_evaluation
all_results, trace_data, metric_data, dataset_used, run_id = run_evaluation(
model="openai/gpt-4",
provider="litellm",
agent_type="both",
enable_otel=True,
hf_token=os.getenv("HF_TOKEN"),
)Lower-level building blocks (generate_dataset_names, push_results_to_hf, compute_leaderboard_row, update_leaderboard, ...) are documented in the Python API reference.
| Dataset | Size | Purpose |
|---|---|---|
kshitijthakkar/smoltrace-tasks (default) |
13 tasks | Quick validation and development |
kshitijthakkar/smoltrace-benchmark-v1 |
132 tasks | Comprehensive evaluation (GAIA, Math, SimpleQA) |
kshitijthakkar/smoltrace-ops-benchmark |
24 tasks | APM / AIOps / SRE / DevOps scenarios |
Copy them to your own account with smoltrace-copy-datasets, or bring your own tasks as a HF dataset or local JSON. Schemas, the ops-benchmark sample-data setup, and custom task creation are covered in the datasets guide.
Community task datasets: the TraceMind-AI collection bundles 40+ ready-to-run task datasets spanning real-world domains — travel, e-commerce, healthcare, finance, DevOps/SRE, AIOps, cybersecurity, and more — for evaluating agents on domain-specific workloads. Point --dataset-name at any of them, e.g. --dataset-name MCP-1st-Birthday/smoltrace-finance-tasks.
--output-format hub # default: 4 datasets on the HF Hub
--output-format json # 5 local JSON files (results, traces, metrics, leaderboard row, metadata)
--output-format opensearch # 4 OpenSearch indexes with typed mappings and bulk indexingDetails and OpenSearch connection options: output formats guide.
Run evaluations on HF cloud hardware without a local GPU (Pro / Team / Enterprise):
hf jobs run \
--flavor cpu-basic \
-s HF_TOKEN=hf_your_token \
-s OPENAI_API_KEY=your_key \
python:3.12 \
bash -c "pip install smoltrace ddgs && smoltrace-eval --model openai/gpt-4 --provider litellm --enable-otel"GPU flavors (t4-small through a100-large), the Python API, and scheduled evaluations are covered in the HF Jobs guide.
Each run creates 3 timestamped datasets, so periodic cleanup keeps your profile tidy:
smoltrace-cleanup --older-than 30d # dry-run preview (default)
smoltrace-cleanup --keep-recent 5 --no-dry-run # actually deleteCleanup is dry-run by default, never touches your leaderboard, and permanently protects the benchmark/tasks datasets. See the dataset management guide.
Every run appends aggregate stats (success rate, avg steps, tokens, duration, cost, CO2, GPU utilization) to your personal leaderboard, with links back to the run's results, traces, and metrics datasets. Community rankings live at huggingface/smolagents-leaderboard — contribute your runs! More in the leaderboard guide.
- Quickstart — first evaluation in minutes
- Configuration — environment variables and options
- Running evaluations — providers, difficulty filters, parallelism
- Agent tools — all 20+ tools with security notes
- GPU metrics & cost — what's collected and how it's aggregated
- Changelog
- llms.txt — machine-readable docs index for LLM agents
- Discord — chat with the community (shared with genai-otel-instrument)
- TraceVerse Community — an open org on the Hub for evaluation & observability datasets, spaces, and real-world MCP apps; join and share your own
- TraceMind-AI collection — 40+ domain-specific task datasets to evaluate against
- GitHub Issues — bug reports and feature requests
Contributions welcome! See the contributing guide.
git clone https://github.com/Mandark-droid/SMOLTRACE.git && cd SMOLTRACE
pip install -e .[dev]
pytestApache-2.0. See LICENSE.
⭐ Star this repo to support Smolagents! Questions? Open an issue.