arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.08463cs.CVcs.AI
收藏

从宽基线立体图像对学习渲染新视角

Learning to Render Novel Views from Wide-Baseline Stereo Pairs

MIT(麻省理工学院)

Yilun Du, Cameron Smith, Ayush Tewari, Vincent Sitzmann

AI总结:

本文针对仅用单个宽基线立体图像对进行新视角合成的难题,提出多视角 Transformer 编码器、图像空间极线采样和轻量交叉注意力渲染器,在大规模真实数据上训练并显著超越现有稀疏观测新视角合成方法。

摘要:

我们提出了一种仅给定单个宽基线立体图像对即可进行新视角合成的方法。在这一具有挑战性的场景中,三维场景点通常只被观测到一次,因此需要基于先验来重建场景几何与外观。我们发现,现有的从稀疏观测进行新视角合成的方法之所以失败,是因为恢复出了错误的三维几何,并且可微渲染的高成本使其难以扩展到大规模训练。为克服这些不足,我们构建了一个多视角 Transformer 编码器,提出了一种高效的图像空间极线采样方案来为目标光线聚合图像特征,并设计了一个轻量级的基于交叉注意力的渲染器。这些贡献使得我们的方法能够在大规模室内外真实世界数据集上进行训练。我们证明,该方法在减少渲染时间的同时,学习到了强大的多视角几何先验。我们在两个真实世界数据集的留出测试场景上进行了广泛比较,在稀疏图像观测的新视角合成任务上显著优于先前工作,并实现了多视角一致的新视角合成。

英文摘要:

We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. In this challenging regime, 3D scene points are regularly observed only once, requiring prior-based reconstruction of scene geometry and appearance. We find that existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry and due to the high cost of differentiable rendering that precludes their scaling to large-scale training. We take a step towards resolving these shortcomings by formulating a multi-view transformer encoder, proposing an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray, and a lightweight cross-attention-based renderer. Our contributions enable training of our method on a large-scale real-world dataset of indoor and outdoor scenes. We demonstrate that our method learns powerful multi-view geometry priors while reducing the rendering time. We conduct extensive comparisons on held-out test scenes across two real-world datasets, significantly outperforming prior work on novel view synthesis from sparse image observations and achieving multi-view-consistent novel view synthesis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑