Part 3: Student Distillation and Finetuning

This chapter covers DAgger/BC setup, visual encoder choices, and post-distillation reinforcement learning.

April 17, 2026 Aditya Dutt 13 min read
robotics computer-vision sim2real

Distillation objective

The student learns from RGB plus proprioception by imitating teacher actions on matched trajectories. A typical objective minimizes action prediction error while preserving temporal consistency.

\[\min_{\phi} \; \mathbb{E}_{(x_t, a_t^*)} \left[\lVert \pi_{\phi}(x_t) - a_t^* \rVert_2^2 \right]\]

Where:

Failure mode to watch. Offline imitation on narrow trajectories can create compounding errors. Use online data aggregation to reduce drift.

DAgger and behavior cloning

A practical recipe:

  1. collect teacher rollouts in diverse randomized simulation conditions,
  2. train the student with behavior cloning,
  3. run the student policy, query teacher labels on visited states (DAgger),
  4. retrain with the expanded dataset.

This loop exposes the student to its own state distribution and improves recovery behavior.

DoorMan three-phase pipeline: teacher RL, student distillation, and student bootstrapping
DoorMan three-phase training flow. Teacher PPO, DAgger distillation, and GRPO bootstrapping form a practical progression for vision-based students.

Visual encoders and policy architecture

For manipulation tasks, architecture choices usually matter as much as loss choice:

Visual randomization examples including image, lighting, material, and camera perturbations
Visual randomization stack. Image-level and renderer-level perturbations improve transfer by exposing the student to broad visual variation during training.
Procedurally generated door variations used in DoorMan training
Task-specific procedural randomization. Door geometry and appearance variation helps the student generalize to previously unseen real-world instances.

Recommendation. Start with a small, reproducible student baseline and only scale encoder complexity after verifying data quality and label alignment.

Post-distillation finetuning

After behavior cloning convergence, optional RL fine-tuning can improve task success and robustness. Typical gains come from:

When fine-tuning, keep safety and smoothness constraints aligned with deployment.

IsaacLab implementation placeholder

# TODO(aditya): Student distillation + finetuning scaffold in IsaacLab

def collect_teacher_dataset(teacher_ckpt, env_cfg):
    # TODO: rollout teacher in randomized sim
    # TODO: save (rgb, proprio, teacher_action, done) tuples
    return "TODO/path/to/dataset"


def train_student_bc(dataset_path):
    # TODO: build encoder + temporal module + action head
    # TODO: behavior cloning training loop
    return "TODO/path/to/student_bc.ckpt"


def dagger_iteration(student_ckpt, teacher_ckpt):
    # TODO: run student policy
    # TODO: query teacher actions on visited states
    # TODO: append to dataset and retrain
    return "TODO/path/to/student_dagger.ckpt"


def finetune_student_with_rl(student_ckpt):
    # TODO: optional PPO/GRPO finetuning with safety constraints
    return "TODO/path/to/student_finetuned.ckpt"
Student policy demo placeholder. Replace with side-by-side clips of teacher policy, distilled student, and post-finetuning behavior.
← Part 2: Teacher Training
Part 4: SysID and Sim2Real Deployment →