VTAO-BiManip: 带物体理解的遮蔽视觉-触觉-动作预训练用于双臂灵巧操作
VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation
- College of Control Science and Engineering, Zhejiang University, Hangzhou, 310027, China（浙江大学控制科学与工程学院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
VTAO-BiManip通过结合视觉-触觉-动作预训练与物体理解，提升双臂灵巧操作的课程强化学习效果，实现超过现有方法20%的成功率。
AI中文摘要：
双臂灵巧操作在机器人领域仍存在重大挑战，原因是每只手的高自由度及其协调性。现有的单手操作技术通常利用人类示范来指导强化学习方法，但无法推广到涉及多种子技能的复杂双臂任务。在本文中，我们引入了VTAO-BiManip，一种新的框架，结合了视觉-触觉-动作预训练与物体理解，以促进课程强化学习，从而实现人般的双臂操作。我们通过纳入手部运动数据来改进先前学习，为双臂协调提供了比二进制触觉反馈更有效的指导。我们的预训练模型利用遮蔽多模态输入预测未来动作以及物体姿态和尺寸，促进跨模态正则化。为了解决多技能学习挑战，我们引入了两阶段课程强化学习方法以稳定训练。我们在瓶盖拧开任务上评估了我们的方法，证明了其在模拟和现实环境中的有效性。我们的方法在成功率上超过了现有视觉-触觉预训练方法超过20%。
英文摘要：
Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination. Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills. In this paper, we introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation. We improve prior learning by incorporating hand motion data, providing more effective guidance for dual-hand coordination than binary tactile feedback. Our pretraining model predicts future actions as well as object pose and size using masked multimodal inputs, facilitating cross-modal regularization. To address the multi-skill learning challenge, we introduce a two-stage curriculum RL approach to stabilize training. We evaluate our method on a bottle-cap unscrewing task, demonstrating its effectiveness in both simulated and real-world environments. Our approach achieves a success rate that surpasses existing visual-tactile pretraining methods by over 20%.
