This repository contains the code accompanying our paper:
Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validation Transactions on Machine Learning Research (TMLR), 2026 Lorenzo Brigato, Stavroula Mougiakakou University of Bern, Switzerland 🇨🇭
The repository provides the experimental framework, datasets, models, and hyperparameter optimization workflows used to conduct the empirical study presented in the paper. Our work introduces Twin (Tune without Validation), a simple and effective validation-free hyperparameter selection pipeline for deep homogeneous classifiers. Instead of relying on validation sets, Twin exploits:
- margin-maximization dynamics of homogeneous networks
- an empirical scaling law linking training and test loss
This enables practitioners to tune learning rate (LR) and weight decay (WD) without holding out data, avoiding the traditional two-step tuning-and-retraining workflow while preserving competitive performance. Across 37 dataset-architecture configurations spanning natural image classification and medical imaging, Twin achieves a mean absolute error of only 1.28% compared to an Oracle that selects hyperparameters using test accuracy.
For additional details, please refer to our manuscript.
This repository contains:
- model implementations
- dataset preparation utilities
- hyperparameter search pipelines
- hyperparameter selection procedures (twin, m-sharpness, oracle, validation set)
Twin is evaluated on a broad set of image classification benchmarks spanning natural images, fine-grained recognition, document analysis, remote sensing, and medical imaging.
The repository includes implementations of the following classifiers:
- Multi-Layer Perceptrons (MLPs)
- ResNet family
- ResNeXt family
- Wide ResNet family
- Compact Convolutional Transformers (CCT)
- EfficientNet family
For the supported datasets and how to obtain them, please refer to the DS.md file.
twin/
├── twin.py # Twin hyperparameter selection approach
├── grid_search.py # Learning rate and weight decay search
├── hbper_scheduler.py # Early-stopping scheduler for Twin
├── config_loader.py # Experiment configuration utilities
├── evaluate_twin.py # Compare Twin selection with Oracle
├── evaluate_sharpness.py # Compare Sharpness selection with Oracle
├── evaluate_testfromval.py # Compute performance of default selection from Validation set
│
├── datasets/ # Dataset storage directory (datasets should be placed here)
│ ├── cifar10/
│ ├── cifair/
│ └── ...
│
├── src/
│ ├── architectures/ # Neural network architectures
│ │ ├── mlp.py
│ │ ├── cifar_resnet.py
│ │ ├── cifar_resnext.py
│ │ ├── wrn.py
│ │ ├── cct.py
│ │ └── medmnist2d_resnet.py
│ │
│ ├── datasets/ # Dataset wrappers and loaders
│ │ ├── cifar.py
│ │ ├── cifair.py
│ │ ├── tinyimagenet.py
│ │ ├── flowers102.py
│ │ ├── cub.py
│ │ ├── eurosat.py
│ │ ├── clamm.py
│ │ ├── isic.py
│ │ └── medmnist.py
│ │
│ ├── methods/ # Training procedures
│ │ ├── xent.py
│ │ ├── xent_adam.py
│ │ ├── xent_multisteplr.py
│ │ ├── xent_pretrained.py
│ │ ├── xent_strongreg.py
│ │ └── randaugment.py
│ │
│ ├── evaluation.py # Evaluation utilities
│ ├── classifiers.py # Classifier definitions
│ ├── ray_classifier.py # Ray integration
│ ├── utils.py
│ └── viz_utils.py
│
├── DS.md # Dataset download instructions
├── requirements.txt
└── README.md
-
Clone this repository.
git clone https://github.com/lorenzobrigato/twin cd twin -
Create a new Conda environment.
conda create -n twin python=3.8 conda activate twin
-
Install Core Dependencies.
pip install -r requirements.txt
Please refer to DS.md for detailed instructions on dataset preparation and expected directory structure.
By default, datasets are expected to be located in ./datasets/<dataset_name>/
Twin starts from a standard grid search over LR and WD. To run the validation-free pipeline, you first execute a grid search using a scheduler (e.g., FIFO), which performs training of all the LR-WD configurations.
This step produces all the necessary logs (training losses, accuracies, and model norms) required by Twin for selection.
Example
python grid_search.py cifar10 \
--architecture mlp-4-256 \
--method CrossEntropyClassifier \
--rand-shift 4 \
--scheduler fifo \
--train-split trainval \
--test-split test \
--epochs 100 \
--batch-size 50 \
--nelems-lr 10 \
--nelems-wd 10 \
--min-lr 5e-5 \
--max-lr 0.5 \
--min-decay 5e-5 \
--max-decay 0.5 \
--history ./logs_grid_search/cifar10_mlp-4-256_CrossEntropyClassifier/history.json \
--save ./logs_grid_search/cifar10_mlp-4-256_CrossEntropyClassifier/model.pth
For running Twin with the early stopping scheduler, e.g., --scheduler hb25%, the metric to be optimized must be changed to --metric training_loss_avg5.
Note that the script enables also more parallelized training with the --cpus-per-trial and --gpus-per-trial arguments which can be customized depending on your available hardware. More info for the possible argument are displyed when calling python grid_search.py -h.
After completing the grid search, Twin selects the best hyperparameters without using any validation set.
This is done by running the evaluation script, which loads the logs produced by grid_search.py, detects the optimization regime (separable vs non-separable), applies the validation-free selection rule (training loss or weight norm), and finally outputs the selected hyperparameters and final performance estimate.
Example
python evaluate_twin.py --folder ./logs_grid_search/cifar10_mlp-4-256_CrossEntropyClassifier/ --eval-metric val_accuracy
In addition to Twin, the repository supports the upper-bound Oracle, the traditional validation-based, and the Sharpness-based selection pipelines used in the paper.
The baseline validation-based selection requires running two separate grid searches. The first search is used for validation selection and should be performed by training on the training split and evaluating on the validation split, using --train-split train and --test-split val. The second search is used to obtain the Oracle performance and should be performed by training on the full training data and evaluating on the test set, using --train-split trainval and --test-split test. For the Oracle search, the scheduler should be set always to --scheduler fifo to ensure that all hyperparameter configurations are fully trained.
python grid_search.py cifar10 \
--architecture mlp-4-256 \
--method CrossEntropyClassifier \
--rand-shift 4 \
--scheduler fifo \
--train-split train \
--test-split val \
--epochs 100 \
--batch-size 50 \
--nelems-lr 10 \
--nelems-wd 10 \
--min-lr 5e-5 \
--max-lr 0.5 \
--min-decay 5e-5 \
--max-decay 0.5 \
--history ./logs_valgrid_search/cifar10_mlp-4-256_CrossEntropyClassifier/history.json \
--save ./logs_valgrid_search/cifar10_mlp-4-256_CrossEntropyClassifier/model.pth
python grid_search.py cifar10 \
--architecture mlp-4-256 \
--method CrossEntropyClassifier \
--rand-shift 4 \
--scheduler fifo \
--train-split trainval \
--test-split test \
--epochs 100 \
--batch-size 50 \
--nelems-lr 10 \
--nelems-wd 10 \
--min-lr 5e-5 \
--max-lr 0.5 \
--min-decay 5e-5 \
--max-decay 0.5 \
--history ./logs_grid_search/cifar10_mlp-4-256_CrossEntropyClassifier/history.json \
--save ./logs_grid_search/cifar10_mlp-4-256_CrossEntropyClassifier/model.pth
Evaluate Validation-Based Selection
After both searches have completed, the selected hyperparameters from the validation search can be evaluated on the corresponding test-search results using, for example:
python evaluate_testfromval.py \
--folder-path ./logs_valgrid_search/cifar10_mlp-4-256_CrossEntropyClassifier/ \
--folder-path-test ./logs_grid_search/cifar10_mlp-4-256_CrossEntropyClassifier/
Evaluate Sharpness-Based Selection
For the Sharpness-based selection (m-sharpness), after having run the Oracle search (--train-split trainval and --test-split test) and saved logs and models, you can evaluate it calling, for example:
python evaluate_sharpness.py --folder ./logs_grid_search/cifar10_mlp-4-256_CrossEntropyClassifier
If you find this repository useful for your research, please consider citing our paper:
@article{
brigato2026twin,
title={Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validation},
author={Lorenzo Brigato and Stavroula Mougiakakou},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2026},
url={https://openreview.net/forum?id=1SIP2M2HJa},
note={}
}