发表机构
University of Science and Technology of China（中国科学技术大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对预训练世界动作模型，提出WAM-OPD在线策略蒸馏方法，利用前缀加权轨迹重放避免环境交互，提升目标任务性能且保持通用技能。
AI 中文摘要
预训练的世界动作模型（WAMs）在多种机器人操作任务中提供了通用能力，然而在不降低预训练技能的前提下将目标任务性能提升至专家水平仍然具有挑战性。我们探索了针对WAMs的在线策略蒸馏（OPD）方法，并提出了WAM-OPD。WAM-OPD继承了OPD方法的优势，即在学生自身诱导的分布下迁移任务特定教师知识，而非直接将学生拟合到狭窄的任务特定数据分布。然而，在闭环操作中，随着学生策略的演化，观测历史会发生变化，需要新鲜的环境回放以保持在线策略。将OPD应用于WAMs需要重复的数据收集，即使在仿真中成本也很高，在真实机器人上往往不切实际。为避免蒸馏过程中的重复环境回放，我们引入了前缀加权轨迹重放（PWTR）。PWTR使用一个固定轨迹池，主要由初始学生回放组成，并辅以任务特定教师回放以拓宽轨迹覆盖。对于从该池中重放的每条轨迹，PWTR将当前策略条件于连续存储的历史上，以生成新鲜的去噪路径，任务特定教师在这些路径上提供监督。尽管这些去噪路径随着策略演化而刷新，重放的环境轨迹保持固定。因此，PWTR使用代理重要性权重对每个决策的蒸馏损失进行重新加权，这些权重由每个决策前轨迹前缀累积的路径分数推导而来，以缓解历史分布偏移。仿真和真实世界实验表明，在蒸馏过程中无需额外环境交互即可实现任务适应。在两种设置中，WAM-OPD均提升了目标任务性能，同时在未适应任务上保持了接近初始的性能。
英文摘要
Pretrained world action models (WAMs) provide generalist capabilities across diverse robotic manipulation tasks, yet improving target-task performance to an expert level without degrading pretrained skills remains challenging. We explore on-policy distillation (OPD) for WAMs and introduce WAM-OPD. WAM-OPD inherits the advantage of OPD methods that transfer task-specific teacher knowledge under the student's own induced distribution, rather than directly fitting the student to a narrow task-specific data distribution. However, in closed-loop manipulation, the observation histories change as the student policy evolves, requiring fresh environment rollouts to remain on-policy. Applying OPD to WAMs entails repeated data collection, which is costly even in simulation and often impractical on real robots. To avoid repeated environment rollouts during distillation, we introduce prefix-weighted trajectory replay (PWTR). PWTR uses a fixed trajectory pool composed primarily of initial-student rollouts, supplemented with task-specific teacher rollouts to broaden trajectory coverage. For each trajectory replayed from this pool, PWTR conditions the current policy on successive stored histories to generate fresh denoising paths, along which the task-specific teacher provides supervision. Although these denoising paths are refreshed as the policy evolves, the replayed environment trajectories remain fixed. PWTR therefore reweights per-decision distillation losses using proxy importance weights derived from path scores accumulated over the trajectory prefix preceding each decision to mitigate the resulting shift in the history distribution. Simulated and real-world experiments demonstrate task adaptation without additional environment interaction during distillation. In both settings, WAM-OPD improves target-task performance while retaining near-initial performance on tasks excluded from adaptation.
