发表机构
UC Berkeley; Harvard University（加州大学伯克利分校; 哈佛大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出世界运动模型（WMMs），用稀疏SE(3)轨迹统一表示动态场景，通过流匹配实现灵活序列建模，支持多种条件化任务，实验验证其多功能性。
AI 中文摘要
为人工智能体赋予空间智能，需要针对动态三维世界建立一个全面的生成先验。我们提出了世界运动模型（WMMs），通过稀疏的SE(3)位姿轨迹来捕捉“随时间推移，何处发生了什么、正在发生什么以及将要发生什么”。WMMs基于这样一个观察：动态场景中的元素可以很好地近似为一组刚性的SE(3)轨迹，这是4D建模的一种最小但富有表现力的基本单元。这种表示将铰接物体、人体、手-物体交互、分段刚性的场景动态、相机运动，甚至机器人状态和动作统一到一个共享空间中。基于这种表示，我们将这些实体的联合分布视为一个灵活的序列建模问题，利用带有逐令牌噪声水平的流匹配技术。结合用于非序列条件化的上下文令牌机制，该公式支持在任意数量的实体和时间步上进行任意到任意的边缘条件化。诸如未来预测、运动填充、模型预测控制、逆运动学、跨实体重定向以及策略学习等任务，都归结为在同一网络上应用不同的掩码。在3D视觉和机器人的6个不同应用上的实验证明了WMMs的多功能性和灵活性，并表现出强劲的性能。
英文摘要
Equipping artificial agents with spatial intelligence requires a comprehensive generative prior over the dynamic 3D world. We propose World Motion Models (WMMs) that capture "what was, is, and will be where across time" via sparse SE(3) pose trajectories. WMMs are built on the observation that elements of dynamic scenes can be well approximated by a set of rigid SE(3) trajectories, a minimal yet expressive primitive for 4D modeling. This representation unifies articulated objects, human bodies, hand-object interactions, piecewise-rigid scene dynamics, camera motion, and even robot states and actions into a single shared space. Given this representation, we cast the joint distribution of these entities as a flexible sequence modeling problem, utilizing flow-matching with per-token noise levels. Coupled with a context token mechanism for non-sequential conditioning, this formulation supports any-to-any marginal conditioning across an arbitrary number of entities and time steps. Tasks such as future prediction, motion infilling, model-predictive control, inverse kinematics, cross-embodiment retargeting, and policy learning all reduce to the application of different masks over the same network. Experiments on 6 diverse applications of 3D vision and robotics demonstrate the versatility and flexibility of WMMs with strong performance.
CommentsAccepted at NeurIPS 2026 (Spotlight). Url: https://jiahuilei.com/projects/wmm/
