Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation

This repository contains the official implementation of the TMLR paper
"Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation". Paper Link | OpenReview

TL;DR

• Stronger robust teachers do not necessarily yield more robust students; the missing factor is sample-wise adversarial transferability from student to teacher.
• We propose SAAD (Sample-wise Adaptive Adversarial Distillation), which up-weights transferable samples and down-weights non-transferable, high-variance samples using a cheap entropy-based proxy.

---

Follow-up Work

For a theoretical understanding of why robust teachers can fail in adversarial distillation, please see our follow-up work:

Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
Accepted to ICML 2026.
Paper Link | OpenReview | GitHub

Installation

All experiments in this repo were run with:

• Python 3.8
• PyTorch 2.4.1

Install the necessary packages using pip:

pip install torch torchvision numpy wandb

For loading pre-trained teacher models, we rely on RobustBench. Please visit the RobustBench repository for setup and model downloads.

---

Supported Methods

This repository includes implementations of the following adversarial distillation methods:

Method	Paper Title	Reference
ARD	Adversarially Robust Distillation	Link
RSLAD	Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better	Link
AdaAD	Boosting Accuracy and Robustness of Student Models via Adaptive Adversarial Distillation	Link
IGDM	Indirect Gradient Matching for Adversarial Robust Distillation	Link
SAAD (Ours)	Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation	Link

---

Quick Start

Run SAAD with the default configuration:

bash main.sh

Citation

If you find this repository useful, please cite:

@article{lee2026samplewise,
  title={Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation},
  author={Hongsin Lee and Hye Won Chung},
  journal={Transactions on Machine Learning Research},
  issn={2835-8856},
  year={2026},
  url={https://openreview.net/forum?id=ek45VamPCE}
}