Distillation objective
The student learns from RGB plus proprioception by imitating teacher actions on matched trajectories. A typical objective minimizes action prediction error while preserving temporal consistency.
\[\min_{\phi} \; \mathbb{E}_{(x_t, a_t^*)} \left[\lVert \pi_{\phi}(x_t) - a_t^* \rVert_2^2 \right]\]Where:
- $x_t$ includes image observations and robot state,
- $a_t^*$ is the teacher target action,
- and optional sequence models improve long-horizon stability.
Failure mode to watch. Offline imitation on narrow trajectories can create compounding errors. Use online data aggregation to reduce drift.
DAgger and behavior cloning
A practical recipe:
- collect teacher rollouts in diverse randomized simulation conditions,
- train the student with behavior cloning,
- run the student policy, query teacher labels on visited states (DAgger),
- retrain with the expanded dataset.
This loop exposes the student to its own state distribution and improves recovery behavior.
Visual encoders and policy architecture
For manipulation tasks, architecture choices usually matter as much as loss choice:
- Encoder: ResNet-style backbones are strong baselines; larger pretrained encoders can help when visual diversity is high.
- Temporal model: LSTM/GRU improves action stability under partial observability and occlusion.
- Policy head: keep output head simple and stable before adding auxiliary objectives.
- Visual randomization: vary textures, lighting, camera perturbations, and distractors to reduce overfitting.
Recommendation. Start with a small, reproducible student baseline and only scale encoder complexity after verifying data quality and label alignment.
Post-distillation finetuning
After behavior cloning convergence, optional RL fine-tuning can improve task success and robustness. Typical gains come from:
- better contact handling,
- improved precision near terminal states,
- and stronger recovery after perturbations.
When fine-tuning, keep safety and smoothness constraints aligned with deployment.
IsaacLab implementation placeholder
# TODO(aditya): Student distillation + finetuning scaffold in IsaacLab
def collect_teacher_dataset(teacher_ckpt, env_cfg):
# TODO: rollout teacher in randomized sim
# TODO: save (rgb, proprio, teacher_action, done) tuples
return "TODO/path/to/dataset"
def train_student_bc(dataset_path):
# TODO: build encoder + temporal module + action head
# TODO: behavior cloning training loop
return "TODO/path/to/student_bc.ckpt"
def dagger_iteration(student_ckpt, teacher_ckpt):
# TODO: run student policy
# TODO: query teacher actions on visited states
# TODO: append to dataset and retrain
return "TODO/path/to/student_dagger.ckpt"
def finetune_student_with_rl(student_ckpt):
# TODO: optional PPO/GRPO finetuning with safety constraints
return "TODO/path/to/student_finetuned.ckpt"