用于机器人操作的无动作预训练物理自回归模型
Physical Autoregressive Model for Robotic Manipulation without Action Pretraining
AI总结：
针对操作数据稀缺问题，提出物理自回归模型（PAR），利用视频预训练世界知识，无需动作预训练，实现准确视频预测与一致动作轨迹，在ManiSkill上PushCube任务达100%成功率。
摘要：
操作数据的稀缺性促使机器人领域利用其他模态的预训练大模型。在这项工作中，我们基于自回归视频生成模型，提出了一种物理自回归模型（PAR），其中物理标记结合了帧和动作，以表示机器人及其环境的联合演化。PAR利用视频预训练中嵌入的世界知识来理解物理动力学，而无需动作预训练，从而实现准确的视频预测和一致的动作轨迹。它还采用基于DiT的解标记器，将帧和动作建模为连续标记，减轻量化误差并促进相互增强。此外，我们引入了带有逆运动学的因果掩码、并行训练和KV缓存机制，以进一步提高性能和效率。在ManiSkill基准上的实验表明，PAR在PushCube任务上达到了100%的成功率，在其他任务上匹配了动作预训练基线的性能，并准确预测了与动作轨迹紧密对齐的未来视频。这些发现强调了通过从自回归视频预训练迁移世界知识来实现机器人操作的一个有前景的方向。项目页面位于：https://hcplab-sysu.github.io/PhysicalAutoregressiveModel/
英文摘要：
The scarcity of manipulation data has motivated the use of pretrained large models from other modalities in robotics. In this work, we build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR), where physical tokens combine frames and actions to represent the joint evolution of the robot and its environment. PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining, enabling accurate video prediction and consistent action trajectories. It also adopts a DiT-based de-tokenizer to model frames and actions as continuous tokens, mitigating quantization errors and facilitating mutual enhancement. Furthermore, we incorporate a causal mask with inverse kinematics, parallel training, and the KV-cache mechanism to further improve performance and efficiency. Experiments on the ManiSkill benchmark show that PAR achieves a 100\% success rate on the PushCube task, matches the performance of action-pretrained baselines on other tasks, and accurately predicts future videos with tightly aligned action trajectories. These findings underscore a promising direction for robotic manipulation by transferring world knowledge from autoregressive video pretraining. The project page is here: https://hcplab-sysu.github.io/PhysicalAutoregressiveModel/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
