M1 Coursework

Overview

• Multiclass classification on the provided handwritten digit dataset with Optuna sweeps for MLP depth/width, regularisation and schedulers (OptunaExperiment).
• Reproducibility/stability reruns for the best trial (Q2) with a retrained checkpoint.
• Triplet / bounded-triplet latent space experiments for a synthetic binary set and CIFAR-10 with k-NN validation on embeddings (TripletOptunaExperiment).
• Visualisation utilities for metrics, latent spaces and t-SNE projections are shared between notebook_helpers.py and plotting/plotting_functions.py. The full analysis and figures live in solution.ipynb.

Getting Started

python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Data

• Local handwritten dataset: handwritten-dataset-tjh200-{train,test}.npz (10 classes, input size [28, 28, 1]).
• CIFAR-10: downloaded automatically to ./datasets by load_train_dataset("CIFAR10") / load_test_dataset("CIFAR10") (input size [32, 32, 3]).
• Synthetic binary: generated on the fly by construct_binary_synthetic_dataset (2D points).

Pre-trained Weights

• best_model_q1.pt: best stable Optuna trial on the local digit task.
• best_model_retrained_q2.pt: reproducibility rerun of the best trial.
• best_synthetic_triplet_model.pt: bounded-triplet embedding for the synthetic binary dataset.
• best_cifar10_triplet_model.pt: bounded-triplet embedding for CIFAR-10.

Load any of them with:

from model.models import FCModel
model = FCModel.load_from_checkpoint("best_model_q1.pt", device="mps")

Re-running Experiments

Handwritten classification (Optuna)

from experiments.hyperp_search import OptunaExperiment

run_cfg = {
    "max_epochs": 100,
    "num_classes": 10,
    "dataset_name": "local",
    "val_size": 0.2,
    "val_target": "f1_macro",
    "experiment_name": "fc-tpesearch-determ-001",
    "seed": 42,
    "device": "mps",
    "input_size": [28, 28, 1],
}

exp = OptunaExperiment(run_cfg)
if not exp.already_ran():
    exp.run(n_trials=500, n_jobs=1, use_journal=True, use_tpe=True)
best_model_path, best_trial = exp.determine_stable_best(num_to_test=10, num_repeats=10, use_journal=True)

Synthetic binary triplet test

from pathlib import Path
from experiments.hyperp_search import TripletOptunaExperiment

run_cfg = {
    "max_epochs": 20,
    "num_classes": 2,
    "dataset_name": "synthetic_binary",
    "val_size": 0.2,
    "experiment_name": "triplet_loss_synthetic_test",
    "seed": 42,
    "device": "cpu",
    "input_size": [2, 1, 1],
}

hyperparams = {
    "batch_size": 10,
    "lr": 1e-5,
    "optimiser": "adamw",
    "weight_decay": 1e-5,
    "hidden_sizes": [64, 64],
    "dropout": 0.1,
    "scheduler": "none",
    "lr_scheduler": None,
    "triplet_alpha": 0.1,
    "loss": "bounded_triplet",
    "latent_radius": 10.0,
    "latent_penalty_weight": 0.0,
    "latent_l2_normalize": False,
    "embedding_dim": 2,
    "classes_per_batch": 2,
    "max_epochs": run_cfg["max_epochs"],
    "patience": run_cfg.get("patience", 10),
}

exp = TripletOptunaExperiment(run_cfg, hyperparams)
save_path = Path("best_synthetic_triplet_model.pt")
if not exp.already_ran(save_path):
    exp.run(n_trials=1, n_jobs=1, use_journal=True)
    exp.copy_best_to_root(new_name=save_path.name)

CIFAR-10 triplet search

from pathlib import Path
from experiments.hyperp_search import TripletOptunaExperiment
run_cfg = {
    "dataset_name": "CIFAR10",
    "val_size": 0.2,
    "input_size": [32, 32, 3],
    "experiment_name": "triplet_bounded_search_12",
    "device": "mps",
    "max_epochs": 80,
    "patience": 5,
    "seed": 42,
    "num_classes": 10,
}

exp = TripletOptunaExperiment(run_cfg)
save_path = Path("best_cifar10_triplet_model.pt")
if not exp.already_ran(save_path):
    exp.run(n_trials=250, n_jobs=1, use_tpe=True, use_journal=True)
    exp.copy_best_to_root(new_name=save_path.name)

Quick evaluation example

from data.data_handler import load_test_dataset, build_loaders
from train.training import evaluate_imbalanced_multiclass
from model.models import FCModel

model = FCModel.load_from_checkpoint("best_model_q1.pt", device="mps")
X_test, y_test = load_test_dataset("local")
test_loader = build_loaders((X_test, y_test), batch_size=256, shuffle=False)
metrics = evaluate_imbalanced_multiclass(model, test_loader, device="mps", num_classes=10)
print(metrics)

Batch scripts

• python experiments/long_run.py: runs the full Optuna sweep plus stability selection for the handwritten digit classifier (writes checkpoints under checkpoints/fc-tpesearch-determ-001 and Optuna logs under optuna/).
• python experiments/long_run_clustering.py: runs the bounded-triplet Optuna search for CIFAR-10 embeddings (checkpoints under checkpoints/triplet_bounded_search_12).

Visualisation

• Notebook-friendly helpers in notebook_helpers.py (plot_tsne_projection, dataset distribution, training comparisons).
• Publication-ready plots in plotting/plotting_functions.py (confusion matrices, ROC, loss curves, latent-space scatter etc.).
• See solution.ipynb for example usage and combined legends for side-by-side latent space / t-SNE plots.

File Structure

• data/: dataset loading and sampler utilities.
• datasets/: downloaded torchvision datasets (CIFAR-10, MNIST if enabled).
• experiments/: Optuna experiment orchestration (hyperp_search.py).
• model/: model definitions (FCModel).
• plotting/: plotting functions used by the notebook and scripts.
• train/: training loops, losses, callbacks, optimisers.
• solution.ipynb: full walkthrough, figures, and all experiment runs.
• checkpoints/: Optuna outputs when re-running searches.
• best_*.pt: saved submission weights (see above).

Rerun Notes

If you want to rerun from scratch, delete the corresponding saved model and either use a new experiment_name in the run config or remove the matching Optuna journal/database under optuna/.