OMeter

Benchmark and compare Ollama models across local and cloud endpoints with rich, sortable tables.

Features

• 📋 List models from local and cloud Ollama endpoints
• 📊 Rich tables with sorting by name, size, context length, modification date, TTFT, or TPS
• 🔃 Reverse sort with --reverse
• ⏱️ Benchmark time-to-first-token (TTFT) and tokens-per-second (TPS)
• 🔍 Model filtering by exact name or family match (e.g. llama3 matches llama3:latest)
• 📤 Export results to JSON or CSV (stdout or file)
• 🧪 Multi-prompt averaging — 3 prompts per model for robust stats
• 🧬 Embedding model support — automatically uses /api/embed for local embedding models
• 🎨 Beautiful CLI powered by rich + InquirerPy

Preview

Cloud model listing — ometer --cloud

Local model listing — ometer --local

Benchmark with per-run breakdown — ometer --local --ttft --tps --verbose --runs 2 --parallel 1

Installation

Install as a uv tool (recommended)

From the project directory:

uv tool install .

Or install directly from GitHub:

uv tool install git+https://github.com/EndoTheDev/OMeter.git

This installs ometer and ometer globally, so you can run them from anywhere.

Update:

uv tool install --upgrade ometer

Uninstall:

uv tool uninstall ometer

Install into a project

uv add ometer

Or via pip:

pip install ometer

Usage

Show the version:

ometer --version

List models with an interactive menu:

ometer

List local models only:

ometer --local

List cloud models only:

ometer --cloud

List both local and cloud models:

ometer --local --cloud

Benchmark time-to-first-token and tokens-per-second:

ometer --cloud --ttft --tps

Benchmark models in parallel for faster results (default is 1 — max 10):

ometer --cloud --ttft --tps --parallel 4

Show per-run breakdown in the table:

ometer --cloud --ttft --tps --verbose

Run with fewer benchmark prompts for faster results (default is 3 — max 3):

ometer --cloud --ttft --tps --verbose --runs 1
ometer --cloud --ttft --tps --verbose --runs 2

Filter to specific models (exact name or family match, accepts multiple names):

ometer --model llama3 --ttft --tps
ometer --local --model llama3.2:3b llama3.3:8b --ttft --tps

Sort results by model size (largest first) or name (A–Z):

ometer --cloud --sort size
ometer --cloud --sort name

Sort by context length (largest first) or modification date (newest first):

ometer --cloud --sort ctx
ometer --local --sort modified

Sort by benchmark metrics — TTFT (lowest/best first) and TPS (highest/best first):

ometer --cloud --ttft --tps --sort ttft
ometer --cloud --ttft --tps --sort tps

Reverse any sort order (worst first, Z–A, oldest first):

ometer --cloud --sort name --reverse
ometer --cloud --ttft --tps --sort tps --reverse

Export results as JSON (to stdout or a file):

ometer --cloud --ttft --tps --json
ometer --cloud --ttft --tps --json results.json

Export results as CSV (to stdout or a file):

ometer --local --ttft --tps --csv
ometer --local --ttft --tps --csv results.csv

See all options:

ometer --help

Environment Variables

OMeter looks for a .env file in this order, using the first one found:

1. ./.env — current working directory (project-specific)
2. ~/.env — home directory (global fallback)
3. ~/.config/ometer/.env — dedicated config directory (recommended for global installs)

Create the config directory and file:

mkdir -p ~/.config/ometer
cat > ~/.config/ometer/.env << 'EOF'
OLLAMA_CLOUD_BASE_URL=https://ollama.com
OLLAMA_CLOUD_API_KEY=your_api_key_here
OLLAMA_LOCAL_BASE_URL=http://localhost:11434

# Number of benchmark prompts per model (1–3, default 3)
OMETER_RUNS=3

# Number of models benchmarked in parallel (default 1, max 10)
OMETER_PARALLEL=1
EOF

The cloud API key is only needed for benchmarking cloud models.

OMeter has five modules that handle distinct concerns:

User ──► cli.py ──► config.py ──► api.py ──► display.py
           │             │            │            │
      arg parsing    .env load    HTTP calls    rich tables
      mode resolve   validate     benchmark     color thresholds
      interactive     clamp       stream        live updates
      export             │            │            │
                         │            │        export.py
                         │            │            │
                         │            │       JSON/CSV output

• cli.py — Entry point, argument parsing, interactive model selection, export dispatch
• config.py — Hierarchical .env loading, settings validation and clamping
• api.py — HTTP communication with Ollama, TTFT/TPS measurement
• display.py — Rich terminal UI, live table updates, percentile-based color coding
• export.py — JSON/CSV export formatting and file output

For detailed documentation, see the docs directory:

• Architecture — Module decomposition, request lifecycle, data entities
• Benchmarking Pipeline — TTFT/TPS methodology, concurrency, color thresholds
• Configuration — Environment variables, CLI flags, loading order
• API Reference — Ollama endpoints, function reference, BenchmarkResult
• Development — Dev setup, running tests, project structure, conventions

License

MIT License — see LICENSE for details.

---

Made by EndoTheDev