智能体世界动作模型
Agentic World Action Models
AI总结:
本文提出AW-0智能体世界动作模型框架,融合代码规划与视频动作预测,经RoboTwin 2.0等任务评估,性能优于VLA、WAM基线,提升了语言遵循等能力。
摘要:
现有世界动作模型可为机器人操作提供视觉前瞻与未来预测,但缺乏显式的组合技能复用机制;智能体机器人技术结合了基于代码的规划与推理及可复用技能,却常缺少对物理动力学的预测依据。本文提出AW-0,一种智能体世界动作模型框架,将基于代码的规划与联合视频及动作预测相融合。我们在统一的混合Transformer架构中,将代码、未来视频与机器人动作建模为三个耦合的生成流:其中代码专家生成参数化技能例程与控制逻辑,视频与动作专家生成未来视觉状态与低级运动指令。代码兼具双重互补作用:一是在世界模型中构建多步预测与控制结构,二是将技能表示为可复用程序供后续智能体执行。我们还提出一种自动整理流水线,该流水线用对齐的程序标记机器人与人类视频轨迹,并将重复例程整合为可复用技能库。在RoboTwin 2.0及物理双臂操作任务上的评估显示,与VLA和WAM基线相比,本文方法的任务性能有所提升,同时在语言遵循、语义推理与鲁棒性方面也有增益,证明智能体代码可实现世界动作模型中规划、预测与执行的连接。
英文摘要:
Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。