Cloud AI Benchmarking Portfolio

Reproducible benchmarking framework for evaluating AI, machine learning, and large language model workloads across AWS, Microsoft Azure, Google Cloud Platform, Docker, and local reference systems.

This repository is the public engineering edition of an accepted bachelor's thesis. It preserves the implementation, methodology, representative results, Docker assets, and research conclusions while presenting the project as a clean GitHub portfolio for AI infrastructure, ML engineering, cloud engineering, and DevOps review.

Project Overview

Cloud AI performance is rarely determined by GPU model alone. CPU frequency, virtualization, input pipelines, Docker overhead, model loading, memory pressure, and region placement can all influence observed results. This project evaluates those effects by running four representative workloads in controlled environments and comparing repeated execution distributions rather than isolated one-off timings.

The project answers a practical engineering question:

How do different classes of AI workloads behave when the same benchmark methodology is deployed across public cloud infrastructure?

Why This Is Technically Interesting

• It compares CPU-bound, mixed CPU/GPU, transformer training, and LLM inference workloads instead of relying on a single synthetic benchmark.
• It uses Docker to control software environments across cloud providers and local baselines.
• It collects execution timing, CPU/RAM telemetry, GPU/VRAM telemetry, configuration metadata, and representative raw outputs.
• It treats variability as a first-class result using repeated runs, medians, IQR, coefficient of variation, and outlier-aware analysis.
• It exposes that provider ranking depends on workload class: CPU-sensitive workloads, transformer training, and LLM inference did not produce the same winner.

Research Questions

1. How do execution-time distributions of representative AI workloads differ across public cloud providers?
2. How stable are execution times across repeated runs on the same virtual machine?
3. How do cloud execution-time distributions compare to local Docker and bare-metal reference systems?
4. To what extent do workload characteristics influence sensitivity to infrastructure-induced variability?

Key Contributions

Contribution	Engineering value
Multi-workload benchmark suite	Covers Random Forest, CNN, BERT fine-tuning, and LLaMA-2 inference.
Cloud comparison methodology	Runs infrastructure-as-a-service experiments on AWS, Azure, and GCP.
Dockerized workload contexts	Keeps dependencies and runtime behavior reproducible across environments.
Telemetry-aware result capture	Records timing, CPU, RAM, GPU, VRAM, configuration, and platform metadata.
Distribution-aware analysis	Prioritizes repeated-run behavior over single best-case benchmark numbers.
Portfolio-ready research artifact	Connects accepted thesis research to maintainable engineering documentation.

Repository Architecture

flowchart LR
    subgraph Inputs
        configs["configs/\nbenchmark parameters"]
        scripts["scripts/\nautomation helpers"]
    end

    subgraph Workloads
        rf["Random Forest\nCPU-bound"]
        cnn["CNN\nmixed CPU/GPU"]
        bert["BERT\ntransformer training"]
        llama["LLaMA-2\nLLM inference"]
    end

    subgraph Runtime
        docker["Docker contexts"]
        cloud["AWS / Azure / GCP"]
        local["Local Docker / bare metal"]
    end

    subgraph Outputs
        raw["representative raw outputs"]
        telemetry["CPU/RAM/GPU telemetry"]
        analysis["analysis scripts"]
        figures["figures and summary tables"]
        docs["GitHub documentation"]
    end

    configs --> rf
    configs --> cnn
    configs --> bert
    configs --> llama
    scripts --> docker
    rf --> docker
    cnn --> docker
    bert --> docker
    llama --> docker
    docker --> cloud
    docker --> local
    cloud --> raw
    local --> raw
    raw --> telemetry
    telemetry --> analysis
    analysis --> figures
    figures --> docs

Loading

Repository Structure

benchmarks/                 Workload entry points
configs/                    Reproducible benchmark configurations
docker/                     Self-contained Docker contexts for each workload
docs/                       Architecture, methodology, reproducibility, and workload guides
examples/                   Minimal smoke-test commands
results/                    Curated figures, summary tables, and representative raw outputs
scripts/                    Analysis, cloud, and automation helpers
src/cloud_ai_benchmarking/  Lightweight package utilities and result schema helpers
tests/                      Repository integrity tests

Technology Stack

Area	Technologies	Role in project
Core language	Python, pandas, NumPy	Benchmark execution, data processing, analysis.
Machine learning	scikit-learn, PyTorch, torchvision	Random Forest and CNN workloads.
Transformers and LLMs	Hugging Face Transformers, datasets, BERT, LLaMA-2, bitsandbytes	Transformer fine-tuning and LLM inference.
Acceleration	CUDA, NVIDIA T4, local RTX-class GPU	GPU execution and telemetry collection.
Cloud platforms	AWS, Microsoft Azure, Google Cloud Platform	IaaS benchmark targets.
Reproducibility	Docker, configuration JSON, metadata outputs	Controlled runtime environments and repeatable runs.
Automation	PowerShell, Bash, Docker CLI	Cloud and local benchmark orchestration.
Visualization	matplotlib, seaborn	Figures, summary tables, and result interpretation.

Cloud Platforms

Platform	Representative environment	Role in thesis
AWS	g4dn.xlarge, NVIDIA T4	Strong final BERT and LLaMA cloud results.
Azure	Standard_NC4as_T4_v3, NVIDIA T4	Strong final Random Forest and CNN cloud results.
GCP	n1-standard-4 with NVIDIA T4	Cloud comparison plus regional/zone variability analysis.
Local Docker	Workstation container baseline	Measures container overhead and local reproducibility.
Local bare metal	Native workstation execution	Reference baseline outside cloud virtualization.

AI Workloads

Workload	Dataset/input	Profile	Main sensitivity
Random Forest	MNIST subset	CPU-bound ML	CPU scheduling, core behavior, virtualization.
CNN	CIFAR-10	Mixed CPU/GPU	Data loading, preprocessing, GPU utilization.
BERT fine-tuning	IMDB sentiment	Transformer training	GPU saturation, tokenization, training stability.
LLaMA-2 inference	Fixed prompts	LLM inference	VRAM pressure, token throughput, generation variability.

Docker Architecture

Each workload has a self-contained Docker context:

flowchart TD
    A["docker/random_forest"] --> A1["Dockerfile"]
    A --> A2["run_random_forest.py"]
    A --> A3["config_*.json"]
    B["docker/cnn"] --> B1["Dockerfile"]
    B --> B2["run_cnn_refactored.py"]
    B --> B3["config_*.json"]
    C["docker/bert"] --> C1["Dockerfile"]
    C --> C2["run_bert.py"]
    C --> C3["config_*.json"]
    D["docker/llama"] --> D1["Dockerfile"]
    D --> D2["run_llama.py"]
    D --> D3["config_*.json"]

Loading

Docker was used to control user-space dependencies across cloud providers. GPU workloads require NVIDIA Container Toolkit on the host.

Experimental Design

sequenceDiagram
    participant Config as Configuration
    participant Runner as Workload Runner
    participant Runtime as Docker/Cloud Runtime
    participant Monitor as Telemetry Monitor
    participant Output as CSV + JSON Output
    participant Analysis as Analysis Scripts

    Config->>Runner: Load CLI flags or config JSON
    Runner->>Runtime: Initialize dataset/model/runtime
    Runner->>Monitor: Start CPU/RAM/GPU sampling
    loop Repeated benchmark runs
        Runner->>Runtime: Execute workload
        Runtime-->>Runner: Timing and task metrics
    end
    Monitor-->>Output: Resource telemetry
    Runner-->>Output: Run records and metadata
    Output->>Analysis: Aggregate distributions
    Analysis-->>Analysis: Median, IQR, CV, outlier checks

Loading

Benchmark Methodology

The accepted thesis emphasized reproducible repeated runs:

• identical workload parameters per comparison;
• Docker-based software control where appropriate;
• repeated execution on cloud and local systems;
• timing plus resource telemetry;
• metadata capture for platform, instance type, runtime, Python version, GPU identity, CPU count, and RAM;
• distribution-aware analysis instead of single-run peak performance.

Statistical Analysis

The analysis uses:

• median execution time;
• mean and standard deviation where useful;
• coefficient of variation for stability;
• interquartile range and outlier detection;
• confidence intervals for selected comparisons;
• cost-performance interpretation based on measured execution time.

Representative summary data lives in results/summary_tables, and representative raw schemas live in results/representative_raw_runs.

Installation

git clone https://github.com/Marcsgarden/cloud-ai-benchmarking
cd cloud-ai-benchmarking
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e ".[dev]"

On Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e ".[dev]"

GPU workloads require a CUDA-capable environment and a compatible PyTorch build. LLaMA-2 also requires Hugging Face access to meta-llama/Llama-2-7b-chat-hf and an HF_TOKEN environment variable.

Quick Start

Run a CPU-only smoke test:

python benchmarks/random_forest/run_random_forest.py --runs 2 --samples 5000 --estimators 50 --platform local --instance-type smoke

Build a Docker image:

docker build -t cloud-ai-benchmark-rf docker/random_forest
docker run --rm cloud-ai-benchmark-rf python /app/run_random_forest.py --runs 2 --platform docker --instance-type local-docker

Run repository integrity tests:

pytest

Running Benchmarks

Random Forest:

python benchmarks/random_forest/run_random_forest.py --runs 5 --samples 15000 --estimators 100 --platform local --instance-type local

CNN:

python benchmarks/cnn/run_cnn_refactored.py --runs 3 --epochs 1 --batch-size 128 --platform local --instance-type local

BERT:

python benchmarks/bert/run_bert.py --runs 3 --epochs 1 --platform local --instance-type local

LLaMA:

export HF_TOKEN=your_hugging_face_token
python benchmarks/llama/run_llama.py --runs 3 --num-prompts 10 --max-new-tokens 100 --platform local --instance-type local

Example Outputs

Benchmark runs write:

• *_results.csv: per-run timing and workload metrics;
• *_metadata.json: platform, hardware, dependency, telemetry, and summary metadata;
• *_config.json: resolved benchmark parameters.

Representative outputs are retained under results/representative_raw_runs so reviewers can inspect schemas without downloading full historical logs.

Representative Figures

These figures summarize the accepted thesis result set. Additional figures and table exports are available in results/figures.

Results Summary

The final thesis found that cloud ranking depends strongly on workload characteristics:

Workload	Final cloud leader	Interpretation
Random Forest	Azure	CPU-sensitive workload benefited from stronger observed CPU behavior.
CNN	Azure	Mixed CPU/GPU training exposed input-pipeline and CPU-frequency effects.
BERT fine-tuning	AWS	Transformer training was GPU-dominated and stable on the tested AWS setup.
LLaMA-2 inference	AWS	Token throughput favored AWS in the accepted final results.

Key Findings

• The fastest cloud provider was not universal across workload classes.
• Nominally GPU-oriented workloads can still be affected by CPU and input-pipeline behavior.
• Local baselines made Docker and cloud overhead visible.
• Repeated-run stability changed the interpretation of benchmark results.
• Availability zone and regional placement can affect observed cloud distributions.

See docs/results.md for more detail.

For interview-oriented project framing, see docs/reviewer-guide.md.

Lessons Learned

• AI infrastructure benchmarking should compare distributions, not single timings.
• GPU model matching is not enough to guarantee comparable behavior across providers.
• Docker improves reproducibility but does not eliminate infrastructure variability.
• Workload selection matters: classical ML, CNNs, transformers, and LLMs stress different bottlenecks.
• Public repositories are stronger when research artifacts are separated from private working-directory clutter.

Future Work

• Add automated Docker build validation in CI for CPU-only contexts.
• Add optional GitHub Actions jobs for documentation and link checking.
• Expand cloud instance coverage beyond NVIDIA T4-class systems.
• Add more LLM serving metrics such as time-to-first-token and prompt-length sensitivity.
• Publish a release artifact containing the portfolio thesis PDF and curated result bundle.

Full Research Paper

The public portfolio edition of the accepted thesis is available here:

docs/thesis/Thesis_Portfolio_Edition.pdf

The submitted university PDF is intentionally not committed because it contains submission-specific metadata.

Citation

Citation metadata is provided in CITATION.cff.

title: "Evaluating Performance of ML/AI/LLM Algorithms in Public Cloud Environments"
author: "Shane Argilagos"
year: 2026
type: thesis

License

This engineering repository is released under the MIT License. The included portfolio thesis PDF remains an academic research document by the author.