面向机器人操作的几何感知4D视频生成
Geometry-aware 4D Video Generation for Robot Manipulation
AI总结：
本文提出了一种几何感知的4D视频生成模型，通过跨视角点图对齐进行训练，以确保生成视频在多视角下的3D一致性，从而在单个RGB-D图像输入下生成时空一致的未来视频序列，并在不依赖相机姿态的情况下实现稳定的视觉和空间对齐预测。
摘要：
理解并预测物理世界的动态可以增强机器人在复杂环境中的规划和交互能力。尽管最近的视频生成模型在建模动态场景方面显示出强大的潜力，但生成在不同摄像机视角下既时间一致又几何一致的视频仍然是一项重大挑战。为此，我们提出了一种4D视频生成模型，通过在训练过程中使用跨视角点图对齐来监督模型，以确保生成视频的多视角3D一致性。通过这种几何监督，模型学习了一个共享的3D场景表示，使其能够从单个RGB-D图像输入中，根据新的视角生成时空一致的未来视频序列，而无需依赖相机姿态作为输入。与现有基线方法相比，我们的方法在多个模拟和现实世界机器人数据集上产生了更稳定和空间对齐的预测。我们进一步表明，预测的4D视频可用于使用现成的6自由度姿态跟踪器恢复机器人末端执行器轨迹，从而生成在新相机视角下具有良好泛化能力的机器人操作策略。
英文摘要：
Understanding and predicting dynamics of the physical world can enhance a robot's ability to plan and interact effectively in complex environments. While recent video generation models have shown strong potential in modeling dynamic scenes, generating videos that are both temporally coherent and geometrically consistent across camera views remains a significant challenge. To address this, we propose a 4D video generation model that enforces multi-view 3D consistency of generated videos by supervising the model with cross-view pointmap alignment during training. Through this geometric supervision, the model learns a shared 3D scene representation, enabling it to generate spatio-temporally aligned future video sequences from novel viewpoints given a single RGB-D image per view, and without relying on camera poses as input. Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets. We further show that the predicted 4D videos can be used to recover robot end-effector trajectories using an off-the-shelf 6DoF pose tracker, yielding robot manipulation policies that generalize well to novel camera viewpoints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
