A compact, controlled study of frozen ImageNet ResNet18 and frozen DINOv2 ViT-S/14 for single-step robot action regression.
This repository is a portfolio-scale representation study, not a closed-loop robot-policy benchmark. It answers a narrower question: under a fixed episode split, frozen encoder, proprioceptive input, and MLP probe, how do an ImageNet baseline and a foundation visual representation behave as training data changes?
On a small, fixed subset of lerobot/aloha_sim_transfer_cube_human:
- With image + state, DINOv2 reached 0.0298 validation MAE versus 0.0409 for ResNet18.
- With only two training episodes, DINOv2 also had lower mean MAE across matched episode subsets: 0.0701 ± 0.0118 versus 0.0808 ± 0.0071.
- Image-only prediction was materially worse than state-only or fused input. This is evidence that visual features contain action-relevant signal in this task; it is not evidence that vision replaces robot state.
- A single brightness perturbation degraded both methods. The one condition tested does not support a general robustness claim.
See docs/RESULTS.md, the plots in results/figures/, and docs/INTERVIEW_QA.md for the careful interpretation.
| Item | Fixed choice |
|---|---|
| Dataset | lerobot/aloha_sim_transfer_cube_human, revision 6a43d500f101255823a9d2b9dc244eeb01a2cd31 |
| Observation / target | top RGB image + 14-D proprioception → 14-D action |
| Split | Episode-level; validation episodes [1, 5, 11, 17] never enter training |
| Encoders | Frozen ImageNet ResNet18; frozen DINOv2 ViT-S/14 |
| Probe | 512 → 256 GELU MLP; AdamW; 1,200 update steps; same seed/settings per matched condition |
| Core metrics | Validation MSE, MAE, per-action-dimension MAE |
| Data efficiency | 2/4/8/16 training episodes (12.5/25/50/100% of the 16-episode train pool), matched subsets across encoders |
No Project 1 dataset was restored. The locally committed repository contains no robot images, video, weights, checkpoints, or feature caches.
The CSVs are the immutable aggregated outputs of the completed run. Recreate all figures locally:
python scripts/plot_results.pyThe source modules implement the frozen probe, episode-level split, metrics, and a checkpoint conversion helper. The original GPU feature extraction/training was deliberately kept outside this repository so it could not modify the concurrently protected OpenPI project. Before a fresh run, create an isolated environment and point a data loader at an approved small LeRobot subset; do not reuse or alter the OpenPI runtime.
configs/mvp.yaml experiment contract
src/ probe, split, metrics, DINOv2 conversion helper
results/*.csv aggregated numerical results
results/figures/*.png reproducible portfolio figures
scripts/plot_results.py figure generator
docs/RESULTS.md protocol, results, and limitations
docs/PROJECT_NOTES.md detailed learning / implementation record
docs/INTERVIEW_QA.md interview preparation
MEMORY.md operational constraints and decisions
- This is a 20-episode probing subset, not a full ALOHA evaluation.
- Each repeated data-efficiency condition resamples episode subsets, not independent simulator seeds or robot rollouts.
- DINOv2 checkpoint conversion was executed against the installed implementation; a future production-quality rerun should log explicit state-dict key coverage and exact package hashes.
- Encoder latency and peak VRAM were not separately microbenchmarked, so this repository intentionally does not invent those numbers.
- A frozen-feature action probe measures linear/MLP accessibility of information, not causal understanding, dynamics learning, or closed-loop control success.