发表机构
Carnegie Mellon University（卡内基梅隆大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出策略校准的DAgger方法，利用生成式策略的预测动作分布离线估计噪声，通过部分去噪引导测量闭环误差，在机器人到达任务中无需噪声扫描即可匹配最佳噪声水平性能。
AI 中文摘要
使用模仿学习训练的策略会随时间累积误差，导致机器人漂移出训练分布。现有方法通过收集策略失败或可能失败处的额外数据来缓解这种协变量偏移。第一种方法将机器人置于不安全的条件下，第二种方法则需要选择合适的噪声分布，以在该噪声下收集新的专家演示。我们提出了策略校准的DAgger（Policy-Calibrated DAgger），该方法利用近期生成式策略的特性，通过使用策略自身的预测动作分布来离线估计策略的噪声。我们测量扩散策略在专家轨迹上的观测点处预测动作的离散程度，并测量其相对于记录轨迹的闭环误差。为解决在多模态动作空间中测量误差的问题，我们在闭环控制期间通过部分去噪将策略引导向轨迹，并利用扩散模型的特性将测量误差反归一化，如同我们未进行引导一样。我们在一个机器人需要在杂乱且狭窄的环境中到达发动机杠杆的场景中进行实验，并在3D照片级真实模拟器和2D平面到达环境中展示了结果。我们表明，我们的方法超越了使用无噪声数据集聚合训练的策略，并匹配了事后最佳噪声水平的性能，而无需对噪声水平进行扫描。
英文摘要
Policies trained with imitation learning can accumulate errors over time, causing the robot to drift outside the training distribution. Existing methods mitigate this covariate shift by collecting additional data where the policy fails or is likely to fail. The first places the robot in unsafe conditions and the second requires choosing an appropriate noise distribution to collect new expert demonstrations under that noise. We propose Policy-Calibrated DAgger, a method that makes use of the properties of recent generative policies to estimate the policy's noise offline by using its own predicted action distribution. We measure a diffusion policy's spread of predicted actions at observations along the expert trajectory and measure its closed-loop error relative to a recorded trajectory. To address issues with measuring error in a multimodal action space, we guide the policy towards the trajectory during closed-loop control through partial denoising, and use properties of a diffusion model to unnormalize the measured error as if we did not guide it. We experiment in a scenario where a robot is tasked to reach an engine lever in a cluttered and narrow environment and show results in a 3D photorealistic simulator and a 2D planar reacher environment. We show that our method surpasses policies trained with dataset aggregation without noising and matches the performance of the best noise level in hindsight, without requiring a sweep over noise levels.
Comments8 pages, 5 figures, in review for ICRA 2027
