通过分离前向和逆向动力学预训练实现解耦的机器人学习
Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining
AI总结：
本文提出DeFI框架，通过分离前向和逆向动力学预训练，利用不同数据源解决视觉-动作耦合问题，实现端到端微调，实验表明在CALVIN ABC-D和SimplerEnv上取得最佳性能。
摘要：
视觉-语言-动作（VLA）模型在构建通用机器人方面展现出巨大潜力，但仍然面临二维图像预测与三维动作预测之间的对齐难题。此外，这种视觉-动作耦合的训练方式限制了模型从大规模无动作网络视频数据中学习。为了解决这些问题，我们提出DeFI，一种新的框架，通过分离视觉前向和逆向动力学预训练来利用各自的数据源，其中视频生成和动作预测被解耦。我们引入了通用前向动力学模型（GFDM），在多样化的人类和机器人视频上进行预训练以预测未来，并引入通用逆向动力学模型（GIDM），通过自监督学习从未标记的视频过渡中推断潜在动作。这些模型随后被整合到统一的架构中，用于下游任务的端到端微调。在这种方式下，GFDM和GIDM首先各自发光，然后相互受益。在CALVIN ABC-D和SimplerEnv上的广泛实验表明，DeFI在任务长度、成功率和现实部署中均取得最佳性能，显著优于先前方法。
英文摘要：
Vision-language-action (VLA) models have shown great potential in building generalist robots, but still face a dilemma-misalignment of 2D image forecasting and 3D action prediction. Besides, such a vision-action entangled training manner limits model learning from large-scale, action-free web video data. To address these issues, we propose DeFI, a novel framework that Decouples visual Forward and Inverse dynamics pretraining to exploit respective data sources, wherein video generation and action prediction are disentangled. We introduce the General Forward Dynamics Model (GFDM), pretrained on diverse human and robot videos for future prediction, and the General Inverse Dynamics Model (GIDM), trained via self-supervised learning to infer latent actions from unlabeled video transitions. These models are then integrated into a unified architecture for end-to-end finetuning on downstream tasks. In this manner, GFDM and GIDM first shine separately and then cooperate for mutual benefit. Extensive experiments on CALVIN ABC-D and SimplerEnv demonstrate state-of-the-art performance, with DeFI achieving an average task length of 4.51 for CALVIN, 51.2% success rate on SimplerEnv-Fractal benchmark and 81.3% success rate in real-world deployment, significantly outperforming prior methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
