Embody4D: 面向具身4D世界建模的通用数据引擎
Embody4D: A Generalist Data Engine for Embodied 4D World Modeling
AI总结：
提出Embody4D视频到视频世界模型，通过3D感知合成管道、潜在置信度专家调制和交互注意力机制，将单目机器人视频转换为多视角视频，解决具身智能中视角稀疏问题，提升下游规划与学习性能。
摘要：
具身智能体需要鲁棒且全面的3D时空表示来支持空间推理、操作理解和下游决策。然而，现有的机器人数据通常从固定或稀疏的视角捕获，仅提供部分且依赖视角的观察，这限制了多视角感知和跨视角泛化。鉴于在真实环境中收集额外视角的困难，我们提出Embody4D，一种专为具身场景设计的视频到视频世界模型，通过将单目机器人视频转换为来自灵活目标相机视角的新视角视频来弥合这一观察差距。首先，为解决训练数据稀缺问题，我们引入了一种3D感知的组合合成管道，以策划一个异构数据集，该数据集组合了跨具身形态的机器人手臂与多样背景，促进了广泛泛化。其次，为强制几何稳定性，我们设计了一种潜在置信度感知的专家调制策略，该策略估计扭曲潜在先验的可靠性，并自适应地将区域路由到复制、修复或修补专家，以实现时空一致的4D生成。最后，为增强操作保真度，我们引入了一种交互感知注意力机制，该机制明确关注机器人交互区域。大量实验表明，Embody4D在视觉评估基准上达到了最先进的性能，同时模拟和真实机器人实验进一步证明了其作为鲁棒数据引擎的有效性，能够合成高保真、视角一致的视频，赋能下游机器人规划和学习。
英文摘要：
Embodied agents require robust and comprehensive 3D spatiotemporal representations to support spatial reasoning, manipulation understanding, and downstream decision making. However, existing robot data are typically captured from fixed or sparse viewpoints, providing only partial and view-dependent observations, which limits multi-view perception and generalization across viewpoints. Given the difficulty of collecting additional viewpoints in real-world settings, we propose Embody4D, a dedicated video-to-video world model for embodied scenarios to bridge this observation gap by transforming a monocular robot video into novel-view videos from flexible target camera viewpoints. First, to tackle training data scarcity, we introduce a 3D-aware compositional synthesis pipeline to curate a heterogeneous dataset compositing cross-embodiment robotic arms with diverse backgrounds, promoting broad generalization. Second, to enforce geometric stability, we devise a latent confidence-aware expert modulation strategy, which estimates the reliability of warped latent priors and adaptively routes regions to copy, repair, or inpaint experts for spatiotemporally consistent 4D generation. Finally, to enhance the fidelity of the manipulation, we incorporate an interaction-aware attention mechanism that explicitly attends to the robotic interaction regions. Extensive experiments show that Embody4D achieves state-of-the-art performance on visual evaluation benchmarks, while both simulated and real-world robotic experiments further demonstrate its effectiveness as a robust data engine for synthesizing high-fidelity, view-consistent videos that empower downstream robotic planning and learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
