arXiv 2602.090232026-05-19cs.RO

TwinRL: 基于数字孪生的强化学习用于真实世界机器人操作

TwinRL: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（信息处理国家重点实验室，计算机学院，北京大学）；Simplexity Robotics（Simplexity机器人）；Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科技大学）

Qinwen Xu, Jiaming Liu, Rui Zhou, Shaojun Shi, Nuowei Han, Zhuoyang Liu, Chenyang Gu, Shuo Gu, Yang Yue, Gao Huang, Wenzhao Zheng, Sirui Han, Peng Jia, Shanghang Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出TwinRL框架，通过数字孪生与真实世界协同训练，提升视觉-语言-动作模型在真实世界中的探索效率和收敛速度，实现高成功率和快速收敛。

摘要：

尽管具有强大的泛化能力，视觉-语言-动作（VLA）模型仍然受到专家演示成本高和现实世界交互有限的限制。虽然在线强化学习（RL）显示出前景，但将其应用于真实世界VLA操作受到探索效率低和探索覆盖受限的阻碍。通过系统性的现实世界实验，我们发现在线RL的有效探索空间主要受监督微调（SFT）期间诱导的轨迹分布所限制。受此观察启发，我们提出TwinRL，一种数字孪生-真实世界协同的后训练框架，通过三个阶段扩展和引导RL探索：SFT预热、孪生RL预热和真实世界RL。TwinRL首先从手机捕捉的场景中重建高保真的数字孪生。在SFT阶段，我们引入一种探索空间扩展策略，将轨迹分布的支持扩展到现实演示之外，重塑探索空间以更有效地进行RL。与将孪生视为数据增强工具不同，我们提出一种孪生RL预热策略，使其能够作为真实世界RL的探索引导。具体而言，TwinRL在数字孪生中执行高效的并行RL，生成填充回放缓冲区的交互轨迹，稳定后续真实世界RL学习。这一过程还识别出易失败但信息丰富的配置，使针对人类在回路中的rollouts进一步提高机器人上的效率。在四个任务中，TwinRL在分布内和分布外区域均实现近100%的成功率，比先前的真实世界RL方法快30%以上，仅需20分钟的机器人交互时间。

英文摘要：

Despite strong generalization capabilities, Vision-Language-Action (VLA) models remain constrained by the high cost of expert demonstrations and limited real-world interaction. While online reinforcement learning (RL) has shown promise, its application to real-world VLA manipulation is hindered by low exploration efficiency and restricted exploration coverage. Through systematic real-world experiments, we observe that the effective exploration space of online RL is largely constrained by the trajectory distribution induced during supervised fine-tuning (SFT). Motivated by this observation, we propose TwinRL, a digital twin-real-world collaborative post-training framework that expands and guides RL exploration for VLA models through three stages: SFT warm-up, twin RL warm-up, and real-world RL. TwinRL first reconstructs a high-fidelity digital twin from smartphone-captured scenes. During the SFT stage, we introduce an exploration space expansion strategy that expands the support of the trajectory distribution beyond real demonstrations, reshaping the exploration space for more effective RL. Rather than treating the twin as a data augmentation tool, we propose a twin RL warm-up strategy that enables it to act as an exploration guide for real-world RL. Specifically, TwinRL performs efficient parallel RL in the digital twin to generate interactive trajectories that populate the replay buffer and stabilize subsequent real-world RL learning. This process also identifies failure-prone yet informative configurations, enabling targeted human-in-the-loop rollouts to further improve on-robot efficiency. Across four tasks, TwinRL achieves near-100% success in both in-distribution and out-of-distribution regions, delivering over 30% faster convergence than prior real-world RL methods with only 20 minutes of on-robot interaction.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。