来自Pensieve的回忆：通过从非校准视频中学习实现新视角合成
Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos
AI总结：
针对现有新视角合成模型依赖校准相机或几何先验的问题，提出两阶段训练策略，先在隐空间自监督预训练，再通过3D高斯基元对齐真实3D几何，实现了仅用非校准视频的高质量合成与位姿估计。
摘要：
当前几乎所有最先进的新视角合成与重建模型都依赖校准相机或额外几何先验进行训练，这些前提条件极大限制了它们在海量非校准数据上的适用性。为降低这一要求并释放大规模非校准视频自监督训练的潜力，我们提出一种新颖的两阶段策略，仅从原始视频帧或多视角图像训练视角合成模型，无需提供相机参数或其他先验。第一阶段，我们学习在隐空间中隐式重建场景，不依赖任何显式3D表示。具体而言，我们预测每帧的隐式相机与场景上下文特征，并采用视角合成模型作为显式渲染的代理。该预训练阶段大幅降低了优化复杂度，促使网络以自监督方式学习潜在的3D一致性。但学习到的隐式相机与隐式场景表示与真实3D世界仍存在较大差距。为缩小这一差距，我们引入第二阶段训练，通过显式预测3D高斯基元（3D Gaussian primitives）。我们额外施加显式高斯泼溅（Gaussian Splatting）渲染损失与深度投影损失，使学习到的隐式表示与基于物理的3D几何对齐。如此一来，第一阶段提供强初始化，第二阶段强化3D一致性——两个阶段互补且互利。大量实验证明了我们方法的有效性，与采用校准、位姿或深度信息监督的方法相比，实现了高质量的新视角合成与准确的相机位姿估计。代码可在https://github.com/Dwawayu/Pensieve获取。
英文摘要：
Currently almost all state-of-the-art novel view synthesis and reconstruction models rely on calibrated cameras or additional geometric priors for training. These prerequisites significantly limit their applicability to massive uncalibrated data. To alleviate this requirement and unlock the potential for self-supervised training on large-scale uncalibrated videos, we propose a novel two-stage strategy to train a view synthesis model from only raw video frames or multi-view images, without providing camera parameters or other priors. In the first stage, we learn to reconstruct the scene implicitly in a latent space without relying on any explicit 3D representation. Specifically, we predict per-frame latent camera and scene context features, and employ a view synthesis model as a proxy for explicit rendering. This pretraining stage substantially reduces the optimization complexity and encourages the network to learn the underlying 3D consistency in a self-supervised manner. The learned latent camera and implicit scene representation have a large gap compared with the real 3D world. To reduce this gap, we introduce the second stage training by explicitly predicting 3D Gaussian primitives. We additionally apply explicit Gaussian Splatting rendering loss and depth projection loss to align the learned latent representations with physically grounded 3D geometry. In this way, Stage 1 provides a strong initialization and Stage 2 enforces 3D consistency - the two stages are complementary and mutually beneficial. Extensive experiments demonstrate the effectiveness of our approach, achieving high-quality novel view synthesis and accurate camera pose estimation, compared to methods that employ supervision with calibration, pose, or depth information. The code is available at https://github.com/Dwawayu/Pensieve.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
