LEAP：将稀疏视图3D建模从相机位姿中解放出来
LEAP: Liberate Sparse-view 3D Modeling from Camera Poses
AI总结：
本文提出无相机位姿的稀疏视图3D建模方法LEAP，通过跨场景共享神经体并以特征相似性聚合图像特征生成辐射场，在精度媲美真值位姿方法的同时比PixelNeRF快400倍。
摘要：
相机位姿对于多视图3D建模是否必要？现有方法大多假设可以获得准确的相机位姿。尽管这一假设在密集视图下可能成立，但对于稀疏视图，准确估计相机位姿往往难以实现。我们的分析表明，带噪声的估计位姿会导致现有稀疏视图3D建模方法性能下降。为解决该问题，我们提出LEAP，一种新颖的无位姿方法，由此挑战相机位姿不可或缺这一主流观念。LEAP摒弃基于位姿的操作，并从数据中学习几何知识。LEAP配备一个神经体，该神经体在不同场景间共享，并经过参数化以编码几何和纹理先验。对于每个输入场景，我们以特征相似性驱动的方式聚合2D图像特征，从而更新神经体。更新后的神经体被解码为辐射场，从而能够从任意视点合成新视图。在以物体为中心和场景级数据集上，我们表明，当先前方法使用来自最先进位姿估计器的预测位姿时，LEAP显著优于这些方法。值得注意的是，LEAP的表现与使用真值位姿的先前方法相当，同时运行速度比PixelNeRF快400倍。我们表明，LEAP可泛化到新的物体类别和场景，并且所学知识与对极几何高度相似。项目页面：https://hwjiang1510.github.io/LEAP/
英文摘要：
Are camera poses necessary for multi-view 3D modeling? Existing approaches predominantly assume access to accurate camera poses. While this assumption might hold for dense views, accurately estimating camera poses for sparse views is often elusive. Our analysis reveals that noisy estimated poses lead to degraded performance for existing sparse-view 3D modeling methods. To address this issue, we present LEAP, a novel pose-free approach, therefore challenging the prevailing notion that camera poses are indispensable. LEAP discards pose-based operations and learns geometric knowledge from data. LEAP is equipped with a neural volume, which is shared across scenes and is parameterized to encode geometry and texture priors. For each incoming scene, we update the neural volume by aggregating 2D image features in a feature-similarity-driven manner. The updated neural volume is decoded into the radiance field, enabling novel view synthesis from any viewpoint. On both object-centric and scene-level datasets, we show that LEAP significantly outperforms prior methods when they employ predicted poses from state-of-the-art pose estimators. Notably, LEAP performs on par with prior approaches that use ground-truth poses while running $400\times$ faster than PixelNeRF. We show LEAP generalizes to novel object categories and scenes, and learns knowledge closely resembles epipolar geometry. Project page: https://hwjiang1510.github.io/LEAP/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
