面向可泛化神经辐射场的几何感知重建与融合精炼渲染
Geometry-aware Reconstruction and Fusion-refined Rendering for Generalizable Neural Radiance Fields
AI总结：
针对可泛化NeRF在挑战条件下泛化受限问题，提出由粗到细的GeFu框架，通过自适应代价聚合、空间-视角聚合器和一致性感知融合策略优化几何重建与渲染，在多数据集上达到SOTA。
摘要：
可泛化NeRF旨在为未见场景合成新视角。常见做法包括构建基于方差的代价体用于几何重建，以及编码3D描述符用于解码新视角。然而，由于不准确的几何、次优的描述符和解码策略，现有方法在挑战性条件下表现出有限的泛化能力。我们逐一解决这些问题。首先，我们发现基于方差的代价体表现出失败模式，因为由于遮挡或反射，对应同一点的像素特征在不同视角间可能不一致。我们引入自适应代价聚合（ACA）方法来放大一致像素对的贡献并抑制不一致的像素对。与以往仅将2D特征融合为描述符的方法不同，我们的方法引入空间-视角聚合器（SVA），通过空间和视角间交互将3D上下文融入描述符。在解码描述符时，我们观察到两种现有解码策略在不同领域各有所长，具有互补性。提出一致性感知融合（CAF）策略以利用两者的优势。我们将上述ACA、SVA和CAF整合到一个由粗到细的框架中，称为几何感知重建与融合精炼渲染（GeFu）。GeFu在多个数据集上达到最先进性能。代码可在https://github.com/TQTQliu/GeFu获取。
英文摘要：
Generalizable NeRF aims to synthesize novel views for unseen scenes. Common practices involve constructing variance-based cost volumes for geometry reconstruction and encoding 3D descriptors for decoding novel views. However, existing methods show limited generalization ability in challenging conditions due to inaccurate geometry, sub-optimal descriptors, and decoding strategies. We address these issues point by point. First, we find the variance-based cost volume exhibits failure patterns as the features of pixels corresponding to the same point can be inconsistent across different views due to occlusions or reflections. We introduce an Adaptive Cost Aggregation (ACA) approach to amplify the contribution of consistent pixel pairs and suppress inconsistent ones. Unlike previous methods that solely fuse 2D features into descriptors, our approach introduces a Spatial-View Aggregator (SVA) to incorporate 3D context into descriptors through spatial and inter-view interaction. When decoding the descriptors, we observe the two existing decoding strategies excel in different areas, which are complementary. A Consistency-Aware Fusion (CAF) strategy is proposed to leverage the advantages of both. We incorporate the above ACA, SVA, and CAF into a coarse-to-fine framework, termed Geometry-aware Reconstruction and Fusion-refined Rendering (GeFu). GeFu attains state-of-the-art performance across multiple datasets. Code is available at https://github.com/TQTQliu/GeFu .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
