Part 2: Teacher Training

This chapter covers teacher objective design, early RL algorithm choice, and practical training heuristics.

April 17, 2026 Aditya Dutt 14 min read
robotics reinforcement-learning sim2real

Teacher setup and objective

Teacher training starts with privileged observations and an asymmetric actor-critic setup. The actor receives noised observations to reduce over-reliance on perfect simulation state, while the critic uses richer state to stabilize value estimation.

$$ \pi_{\theta}(a \mid \tilde{o}), \quad \tilde{o} = o + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \sigma^2 I) $$ (1)

This setup prepares the policy for the information gap that appears later during visual distillation.

Why this matters. A teacher that is overly tuned to perfect state can become hard to distill and brittle under deployment noise.

Simulation environment + noise ε full GT Actor obs: proprio + noisy obj + ε Transferred to student Critic obs: full privileged state Discarded after training π(a | õ) — LSTM + MLP Dense skip connections V(o_priv) — MLP GAE advantage estimation Actions a_t Advantage Â_t PPO objective: max E[min(r_t Â_t, clip(r_t, 1±ε) Â_t)]
Asymmetric actor-critic. The actor is trained on noised observations while the critic uses privileged state.

RL algorithm choice (early decision)

Choose the RL algorithm before tuning other details. This decision controls data efficiency, stability, and the kind of policy behavior your teacher converges to.

PPO baseline

PPO remains a reliable default for many manipulation tasks, especially when compute is abundant and you need predictable optimization behavior.

SAC and stability trade-offs

SAC can be sample-efficient but may show instability in contact-rich settings if entropy and critic tuning are not carefully managed.

FlashSAC and improved off-policy training

FlashSAC-style improvements can make off-policy approaches more practical for sim2real, particularly when you want to reuse replay data efficiently.

Recommendation. Start with PPO for a dependable teacher baseline, then test SAC/FlashSAC only after you have stable reward design, reset logic, and action-space choices.

Action space design

Action space is a transfer-critical design choice. For many fixed-base manipulators:

Action smoothing and rate limits are worth adding in simulation during teacher training because they reduce distribution shift in later deployment.

Resets, exploration, and randomization

After algorithm and action-space selection, focus on training robustness:

OmniReset staged reset pipeline with collected reset states and zero-shot transfer
OmniReset pipeline. Collecting diverse reset states broadens coverage and supports robust teacher RL before perception distillation.
DoorMan staged reset buffer across task stages with cached snapshots
DoorMan staged resets. Cached snapshots from later stages are reused to accelerate long-horizon exploration and reduce sparse-reward failure modes.
Reference state initialization examples across simulated manipulation scenes
Reference state initialization. Initializing from diverse demonstration states helps the teacher practice recovery and downstream student robustness.

Practical sequence. Stabilize reward and reset coverage first, then randomization ranges, and only then aggressive hyperparameter sweeps.

IsaacLab implementation placeholder

# TODO(aditya): Teacher training scaffold in IsaacLab
# Replace these placeholders with actual config classes and runner APIs.

def build_teacher_env():
    # TODO: create privileged observation manager
    # TODO: add domain randomization manager
    # TODO: configure reset events (OmniReset style)
    pass


def train_teacher():
    algo = "PPO"  # TODO: compare against SAC / FlashSAC setup
    env = build_teacher_env()
    # TODO: instantiate policy + critic networks
    # TODO: run rollout collection and updates
    # TODO: checkpoint best teacher
    return {"checkpoint": "TODO/path/to/teacher.ckpt", "algo": algo}
Teacher training demo placeholder. Replace with rollout and convergence clips comparing algorithm choices.
← Part 1: Overview
Part 3: Student Distillation and Finetuning →