MESD — black-box model extraction with synthetic data

Research code (MESD) for model stealing / extraction in a synthetic-data setting: use an LLM to write diffusion prompts from class names, generate images, query a black-box victim for (soft) labels, and train a substitute model—ViT with LoRA by default, plus an optional ResNet-34 baseline.

CUB-200-2011 is the primary worked example (fine-grained classes → prompts → PNGs per class → substitute training).

Ethics. For authorized research, education, and defense evaluation only. Do not use this code to attack third-party APIs or services without explicit permission.

---

Overview

Stage	What it does
Synthetic pool	LLM produces JSON prompt lists; a diffusion model renders images into per-class folders (shadow / query pool).
Substitute training	Images are fed to the victim; soft targets train a student (ViT+LoRA or ResNet) to mimic the victim on held-out test data.

You can swap LLM and diffusion checkpoints in scripts/synthesize_cubs200.py (e.g. different open or API models) without changing the rest of the pipeline.

---

Repository layout

Path	Role
scripts/synthesize_cubs200.py	Build the synthetic image pool from classes.txt (default: under save_image/…).
scripts/train_substitute_vit_lora.py	Query victim (black-box dir layout) + train ViT + LoRA substitute.
scripts/extract_substitute_resnet.py	Optional ResNet-34 substitute (lighter baseline).
blackbox_model/	Datasets, model zoo, Blackbox victim loader.
mesd/transferset.py	Batch querying and soft-label transferset construction.
train_utils.py, sampler.py	Training utilities and query / active-selection samplers.

---

Setup

cd <repo-root>
python -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt

GPU is strongly recommended. Some Hugging Face models require license acceptance and huggingface-cli login.

---

Data

Put CUB-200-2011 under data/CUB_200_2011/ (see data/README.md). Synthesis only needs metadata such as classes.txt; evaluation uses the official train/test split.

---

Usage

1) Synthesize images (query pool)

Up to 5 PNGs per class by default:

export PYTHONPATH=$(pwd)
python scripts/synthesize_cubs200.py \
  --classes_file data/CUB_200_2011/classes.txt \
  --output_root save_image/CUBS200_llama3d1_new

2) Train ViT + LoRA substitute

Point --victim_model_dir at a black-box release (params.json + checkpoint). See models/victim/README.md.

export PYTHONPATH=$(pwd)
python scripts/train_substitute_vit_lora.py \
  --victim_model_dir models/victim/your_cubs200_vit_lora \
  --synthetic_glob "save_image/CUBS200_llama3d1_new/*/*.png" \
  --initial_budget 10000 \
  --save_dir outputs/cubs200_vit_lora_run1

Use --pretrained_model_name to switch backbone (e.g. same vs different pretrained ViT for cross-architecture extraction).

3) ResNet-34 substitute (optional)

export PYTHONPATH=$(pwd)
python scripts/extract_substitute_resnet.py \
  --blackbox_dir models/victim/cubs200-resnet34 \
  --image_glob "save_image/CUBS200_llama3d1_new/*/*.png" \
  --initial_budget 2000 \
  --save_dir outputs/cubs200_resnet_rand

Pass --h_dir path/to/victim.pt if you load a custom Victim wrapper instead of Blackbox.from_modeldir.

---

License

Code is released under the MIT License (see LICENSE). Third-party weights and datasets remain under their own terms.