通过仿真引导微调实现策略对真实世界的快速适应
Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning
AI总结：
本文提出仿真引导微调（SGFT）框架，利用仿真中学习的价值函数引导真实世界探索，在五个灵巧操作任务上以最多减少一个数量级的真实样本显著加速策略适应并超越基线方法。
摘要：
机器人学习需要大量高质量数据才能实现泛化的承诺。然而，在真实世界中收集大规模数据集成本高昂。物理仿真器可以廉价地生成覆盖广泛状态、动作和环境的大规模数据集。然而，物理引擎本质上是对现实的有误设定近似。这使得从仿真到现实的直接零样本迁移具有挑战性，尤其是在需要精确且力敏感操作的任务中。因此，利用小规模真实世界数据集对这些策略进行微调，是扩展机器人学习的一条有吸引力的路径。然而，当前的强化学习微调框架采用通用的、非结构化的探索策略，效率过低，难以使真实世界适应变得切实可行。本文提出了仿真引导微调（SGFT）框架，展示了如何从物理仿真器中提取结构先验，以大幅加速真实世界适应。具体而言，我们的方法使用在仿真中学习到的价值函数来引导真实世界探索。我们在五个真实世界灵巧操作任务上展示了这一方法，在这些任务中零样本仿真到现实迁移均失败。我们进一步证明，我们的框架显著优于基线微调方法，所需的真实世界样本最多减少一个数量级，并在先前方法完全失败的困难任务上取得成功。最后但同样重要的是，我们为这一新范式提供了理论论证，支撑了SGFT在面对较大仿真到现实动力学差距时能够快速学习高性能策略的机制。项目网页：https://weirdlabuw.github.io/sgft/{weirdlabuw.github.io/sgft}
英文摘要：
Robot learning requires a considerable amount of high-quality data to realize the promise of generalization. However, large data sets are costly to collect in the real world. Physics simulators can cheaply generate vast data sets with broad coverage over states, actions, and environments. However, physics engines are fundamentally misspecified approximations to reality. This makes direct zero-shot transfer from simulation to reality challenging, especially in tasks where precise and force-sensitive manipulation is necessary. Thus, fine-tuning these policies with small real-world data sets is an appealing pathway for scaling robot learning. However, current reinforcement learning fine-tuning frameworks leverage general, unstructured exploration strategies which are too inefficient to make real-world adaptation practical. This paper introduces the Simulation-Guided Fine-tuning (SGFT) framework, which demonstrates how to extract structural priors from physics simulators to substantially accelerate real-world adaptation. Specifically, our approach uses a value function learned in simulation to guide real-world exploration. We demonstrate this approach across five real-world dexterous manipulation tasks where zero-shot sim-to-real transfer fails. We further demonstrate our framework substantially outperforms baseline fine-tuning methods, requiring up to an order of magnitude fewer real-world samples and succeeding at difficult tasks where prior approaches fail entirely. Last but not least, we provide theoretical justification for this new paradigm which underpins how SGFT can rapidly learn high-performance policies in the face of large sim-to-real dynamics gaps. Project webpage: https://weirdlabuw.github.io/sgft/{weirdlabuw.github.io/sgft}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
