Splatter-360: 面向宽基线全景图像的可泛化360°高斯泼溅
Splatter-360: Generalizable 360$^{\circ}$ Gaussian Splatting for Wide-baseline Panoramic Images
AI总结：
针对宽基线全景图像新视角合成中现有3DGS方法易过拟合的问题，提出Splatter-360框架，通过球面域多视图匹配、3D感知双投影编码器与跨视图注意力实现高质量实时渲染，泛化性能优于现有方法。
摘要：
宽基线全景图像常被用于VR和仿真等应用中，以降低采集人力成本与存储需求。然而，从这类全景图像实时合成新视角仍是一项重大挑战，这主要源于全景图像的高分辨率与固有畸变。现有3D高斯泼溅（3D Gaussian Splatting, 3DGS）方法虽能在窄基线下生成照片级真实感视角，但在处理宽基线全景图像时，由于难以从稀疏的360°视图中学习精确几何结构，往往会对训练视图产生过拟合。\n本文提出了Splatter-360，这是一种专为处理宽基线全景图像设计的新型端到端可泛化3DGS框架。与现有方法不同，Splatter-360通过球面扫描算法构建球面代价体，直接在球面域中执行多视图匹配，增强了网络的深度感知与几何估计能力。此外，我们引入了3D感知双投影编码器以缓解全景图像的固有畸变，并融合跨视图注意力机制以提升多视角间的特征交互，从而实现鲁棒的3D感知特征表示与实时渲染能力。\n在HM3D和Replica数据集上的实验结果表明，在宽基线全景图像的合成质量与泛化性能方面，Splatter-360均显著优于当前最先进的NeRF与3DGS方法（如PanoGRF、MVSplat、DepthSplat和HiSplat）。代码与训练好的模型可在https://3d-aigc.github.io/Splatter-360/获取。
英文摘要：
Wide-baseline panoramic images are frequently used in applications like VR and simulations to minimize capturing labor costs and storage needs. However, synthesizing novel views from these panoramic images in real time remains a significant challenge, especially due to panoramic imagery's high resolution and inherent distortions. Although existing 3D Gaussian splatting (3DGS) methods can produce photo-realistic views under narrow baselines, they often overfit the training views when dealing with wide-baseline panoramic images due to the difficulty in learning precise geometry from sparse 360$^{\circ}$ views. This paper presents \textit{Splatter-360}, a novel end-to-end generalizable 3DGS framework designed to handle wide-baseline panoramic images. Unlike previous approaches, \textit{Splatter-360} performs multi-view matching directly in the spherical domain by constructing a spherical cost volume through a spherical sweep algorithm, enhancing the network's depth perception and geometry estimation. Additionally, we introduce a 3D-aware bi-projection encoder to mitigate the distortions inherent in panoramic images and integrate cross-view attention to improve feature interactions across multiple viewpoints. This enables robust 3D-aware feature representations and real-time rendering capabilities. Experimental results on the HM3D~\cite{hm3d} and Replica~\cite{replica} demonstrate that \textit{Splatter-360} significantly outperforms state-of-the-art NeRF and 3DGS methods (e.g., PanoGRF, MVSplat, DepthSplat, and HiSplat) in both synthesis quality and generalization performance for wide-baseline panoramic images. Code and trained models are available at \url{https://3d-aigc.github.io/Splatter-360/}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
