PointWAM：用于灵巧机器人操作的3D世界动作建模

中文摘要 AI 辅助

世界动作模型联合学习预测世界动态和机器人动作，使得学习到的内部世界动态能够指导准确的动作。现有方法通常将世界表示为RGB帧或潜在对应物，同时将动作预测为末端执行器姿态或关节角度，但它们往往难以捕捉灵巧操作中至关重要的3D空间结构和接触几何。我们引入了点世界动作模型（PointWAM），这是一种3D世界动作模型，它将世界分解为场景（即环境）和手（即执行者），并在共享的时空坐标框架内联合预测两者作为3D点轨迹。这种显式、解耦的表示使得能够在大规模人类演示视频上进行有效的预训练，而无需任何特定于任务的对象或关键点选择。给定彩色点云和语言指令，PointWAM预测场景和手如何在3D空间中随时间共同演化，然后将预测的手部运动重新定向为机器人动作。在人类视频上进行预训练使平均DexJoCo成功率提高了56.9个百分点，而场景轨迹监督比仅预测手部动作增加了10.9个百分点。结合两者，PointWAM在十项DexJoCo任务上超越了先前的最先进水平，提高了11.7个百分点，并在真实机器人上优于强大的VLA模型。

英文摘要

World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.

发表机构

• POSTECH（浦项科技大学）
• KAIST（韩国科学技术院）
• Hanyang University（汉阳大学）
• RLWRLD

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

Preprint. Project page: https://chrockey.github.io/PointWAM