TEMPO: Tracking and Evaluating Metrics for Performance Optimization

A comprehensive system for tracking, analyzing, and visualizing performance metrics across different benchmarks and applications with a modern web interface and API.

flowchart LR
    classDef mainNode fill:#3498db,stroke:#2980b9,color:white,stroke-width:2px,font-weight:bold
    classDef storageNode fill:#2ecc71,stroke:#27ae60,color:white,stroke-width:2px,font-weight:bold
    classDef processNode fill:#9b59b6,stroke:#8e44ad,color:white,stroke-width:2px,font-weight:bold
    classDef inputNode fill:#e67e22,stroke:#d35400,color:white,stroke-width:2px,font-weight:bold
    classDef outputNode fill:#e74c3c,stroke:#c0392b,color:white,stroke-width:2px,font-weight:bold
    classDef userNode fill:#f1c40f,stroke:#f39c12,color:#34495e,stroke-width:2px,font-weight:bold

    %% Main components
    T["TEMPO System"]:::mainNode
    DB[(Performance<br>Database)]:::storageNode
    API{{"REST API"}}:::processNode
    
    %% Input sources
    BM[["Benchmark<br>Scripts"]]:::inputNode
    CI[["CI/CD<br>Pipeline"]]:::inputNode
    MT[["Manual<br>Testing"]]:::inputNode
    
    %% Output destinations
    DASH["Web<br>Dashboard"]:::outputNode
    REP["Analysis<br>Reports"]:::outputNode
    ALERT["Regression<br>Alerts"]:::outputNode
    
    %% Users
    DEV(["Developers"]):::userNode
    RES(["Researchers"]):::userNode
    MGT(["Management"]):::userNode
    
    %% Relationships - Inputs
    BM -->|"Submit<br>Results"| API
    CI -->|"Automated<br>Submissions"| API
    MT -->|"Ad-hoc<br>Testing"| API
    
    %% Core processing
    API -->|"Store<br>Metrics"| DB
    DB --- T
    
    %% Relationships - Outputs
    DB -->|"Serve<br>Data"| DASH
    DB -->|"Generate"| REP
    DB -->|"Detect"| ALERT
    
    %% User interactions
    DEV -.->|"Run Tests<br>View Metrics"| BM & DASH
    RES -.->|"Analyze<br>Trends"| DASH & REP
    MGT -.->|"Review<br>Performance"| REP & ALERT
    
    %% Subgraphs
    subgraph Input ["Data Collection"]
        BM
        CI
        MT
    end
    
    subgraph Core ["Core System"]
        T
        API
        DB
    end
    
    subgraph Output ["Insights & Visualization"]
        DASH
        REP
        ALERT
    end
    
    subgraph Users ["Stakeholders"]
        DEV
        RES
        MGT
    end

Loading

Features

• Structured benchmark result storage with version control integration
• Modern, responsive web interface for intuitive data visualization
• RESTful API for standardized result submission
• SQLite database for reliable, portable storage
• Git integration for version tracking and correlation
• Customizable metric types and parameter grouping
• Filtering and advanced search capabilities
• Detailed view of performance metrics and parameters

System Architecture

TEMPO consists of three main components:

1. Database Server: A Flask-based web application that stores and serves benchmark results
2. Benchmark Scripts: Tools to run benchmarks and automatically log results
3. Web Interface: A user-friendly dashboard for viewing and analyzing results

Setup

Prerequisites

• Python 3.6+
• pip (Python package manager)

Installation

1. Clone the repository
2. Install dependencies:

pip install -r requirements.txt

For API client:

pip install requests

Running the server

python app.py

The server will be available at http://localhost:5040

Database initialization

The database is automatically initialized when you start the server. If you want to add sample data:

flask --app app.py seed-db

Usage

Web Interface

Access the web interface at http://localhost:5040 to:

• View performance runs
• Filter by branch, dataset, or other parameters
• Compare different implementations
• View detailed metrics for each run
• Analyze performance trends over time

Running Benchmarks

TEMPO includes example benchmarks in the example_scripts directory:

cd example_scripts
python3 matrix_benchmark.py ./mock_matmul.py --m 1024 --k 1024 --n 1024 --threads 4 --iterations 3

REST API

Add a new performance run

curl -X POST http://localhost:5040/api/runs \
  -H "Content-Type: application/json" \
  -d '{
    "commit_id": "abcdef1234567890",
    "branch_name": "main",
    "dataset_name": "matrix_1024x1024",
    "bin_name": "matmul_benchmark",
    "base_parameters": {"threads": 4, "precision": "fp64"},
    "metrics": [
      {
        "metric_name": "execution_time",
        "metric_value": 123.45,
        "metric_type": "timer",
        "component_name": "matrix_mult",
        "additional_context": {"unit": "ms"}
      }
    ],
    "parameters": [
      {
        "param_name": "matrix_size",
        "param_value": "1024",
        "param_type": "integer",
        "param_group": "algorithm"
      }
    ]
  }'

Get a specific run

curl -X GET http://localhost:5040/api/runs/1

Command-line Client

The project includes a command-line client for adding data:

# Add data using API
python client/update_client.py --api --commit "abcdef123" --branch "main" --dataset "matrix_1024x1024" --binary "matmul_benchmark"

# Add data using direct SQLAlchemy access
python client/update_client.py --db "instance/perf_data.db" --commit "abcdef123" --branch "main" --dataset "matrix_1024x1024" --binary "matmul_benchmark"

# Add data from JSON file
python client/update_client.py --json data.json

Database Schema

The application uses three main tables:

performance_runs

• Primary key for performance test runs
• Stores commit ID, branch name, dataset, binary, and timestamp

performance_metrics

• Stores individual metrics for each run
• Links to the parent run via foreign key
• Includes metric name, value, type, and additional context

tuning_parameters

• Stores tuning parameters used in each run
• Links to the parent run via foreign key
• Includes parameter name, value, type, and grouping

Benefits

By implementing TEMPO, you can:

• Establish baseline performance metrics
• Track performance evolution over time
• Compare results across different implementations and configurations
• Identify performance regressions early
• Make data-driven optimization decisions

Documentation

For comprehensive documentation, please refer to:

• User Guide - Detailed instructions for using TEMPO
• Example Scripts - Example benchmarks and usage patterns

Customization

• Modify app/templates/ to customize the web interface
• Add new API endpoints in app/views.py
• Extend the data model in app/models.py

License

MIT