发表机构
Tsinghua University; TeleAI, China Telecom; Shenzhen Technology University（清华大学; 中国电信 TeleAI; 深圳技术大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
FutureDuet将主相机与腕部相机解耦，分别设计未来监督目标，通过ActionDiT融合任务与交互状态，在RoboTwin50和LIBERO上显著提升机器人操作成功率。
AI 中文摘要
世界动作模型（WAMs）通过未来视觉监督增强机器人动作生成。现有WAMs通常将主相机和腕部相机的观测融合为一个视觉流，并使用相同的未来视频目标进行训练，尽管它们的视觉动态不同。稳定的主相机揭示场景级别的任务演化，而腕部相机随末端执行器移动，将局部交互变化与视角变化和自遮挡混合在一起。这些对比性的预测需求表明，两个视角可能受益于不同的未来目标。我们提出了FutureDuet，它保留两个视角用于控制，同时允许每个视觉流接收不同的未来目标。对于主视角，未来RGB建模任务演化，而交互掩码和机器人骨架将监督聚焦于任务对象和机器人运动。对于腕部流，未来潜在预测建模短时交互变化，无需像素级重建。ActionDiT联合读取生成的任务状态和交互状态，将场景级进展与近距离交互证据相结合。所有辅助预测模块仅在训练时使用，不增加推理开销。FutureDuet在RoboTwin50上实现了94.2%的干净设置和94.1%的随机化设置成功率，在LIBERO上实现了99.2%的平均成功率。在需要精确交互的六个RoboTwin50任务上改进最为显著，在干净和随机化设置中平均比Fast-WAM高出9.2%和12.8%。对照研究进一步表明，分离腕部路径并为两个视角分别设计未来监督可带来互补性收益。
英文摘要
World Action Models (WAMs) augment robot action generation with future visual supervision. Existing WAMs commonly fuse main and wrist observations into one visual stream and train both with the same future-video objective, despite their different visual dynamics. A stable main camera reveals scene-level task evolution, whereas wrist cameras move with the end effector, mixing local interaction changes with viewpoint shifts and self-occlusion. These contrasting predictive demands suggest that the two views may benefit from different future objectives. We introduce FutureDuet, which retains both views for control, while allowing each visual stream to receive a different future objective. For the main view, future RGB models task evolution, while interaction masks and robot skeletons focus supervision on task objects and robot motion. For the wrist stream, future latent prediction models short-horizon interaction changes without requiring pixel-level reconstruction. ActionDiT jointly reads the resulting Task State and Interaction State, combining scene-level progress with close-range interaction evidence. All auxiliary prediction modules are training-only, adding no inference overhead. FutureDuet achieves 94.2% clean and 94.1% randomized success on RoboTwin50 and 99.2% average success on LIBERO. The improvements are most pronounced on six RoboTwin50 tasks that require precise interaction, averaging gains of 9.2% and 12.8% over Fast-WAM in clean and randomized settings. Controlled studies further show complementary gains from separating the wrist pathway and designing future supervision separately for the two views.
Comments13 pages, 7 figures. Project page: https://1723578110.github.io/futureduet-web/
