JOG3R：迈向3D一致的视频生成器
JOG3R: Towards 3D-Consistent Video Generators
AI总结：
本文提出JOG3R，首个统一视频生成与相机姿态估计的3D一致模型，通过联合训练实现高质量视频生成与姿态估计，可扩展至其他3D感知任务。
摘要：
图像生成器的涌现能力已经催生了许多有影响力的零样本或少样本应用。受此成功的启发，我们研究视频生成器是否同样展现出3D感知能力。以运动恢复结构作为3D感知任务，我们测试视频生成器（本例中为OpenSora）的中间特征是否能支持相机姿态估计。令人惊讶的是，起初我们只发现这两个任务之间存在弱相关性。深入调查揭示，尽管视频生成器产生了看似合理的视频帧，但这些帧本身并非真正3D一致的。相反，我们提出联合训练这两个任务，使用光度生成和3D感知误差。具体来说，我们发现最先进的视频生成和相机姿态估计（即DUSt3R [79]）网络共享共同结构，并提出一种统一两者的架构。所提出的统一模型名为JOG3R，在生成3D一致视频的同时，产生具有竞争力的相机姿态估计质量。总之，我们提出了第一个3D一致、生成逼真视频帧的统一视频生成器，并且该生成器可能被重新用于其他3D感知任务。
英文摘要：
Emergent capabilities of image generators have led to many impactful zero- or few-shot applications. Inspired by this success, we investigate whether video generators similarly exhibit 3D-awareness. Using structure-from-motion as a 3D-aware task, we test if intermediate features of a video generator - OpenSora in our case - can support camera pose estimation. Surprisingly, at first, we only find a weak correlation between the two tasks. Deeper investigation reveals that although the video generator produces plausible video frames, the frames themselves are not truly 3D-consistent. Instead, we propose to jointly train for the two tasks, using photometric generation and 3D aware errors. Specifically, we find that SoTA video generation and camera pose estimation (i.e.,DUSt3R [79]) networks share common structures, and propose an architecture that unifies the two. The proposed unified model, named \nameMethod, produces camera pose estimates with competitive quality while producing 3D-consistent videos. In summary, we propose the first unified video generator that is 3D-consistent, generates realistic video frames, and can potentially be repurposed for other 3D-aware tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
