Feat2GS: 通过高斯点云探测视觉基础模型
Feat2GS: Probing Visual Foundation Models with Gaussian Splatting
AI总结：
Feat2GS通过高斯点云探测视觉基础模型的3D理解能力，无需3D数据即可分析几何和纹理意识。
摘要：
鉴于视觉基础模型（VFMs）是在大量数据集上训练，但通常局限于2D图像，一个自然的问题浮现：它们对三维世界理解有多好？由于架构和训练协议（即目标、代理任务）的不同，需要一个统一的框架来公平且全面地探测它们的三维意识。现有的3D探测工作建议单视图2.5D估计（例如深度和法线）或双视图稀疏2D对应（例如匹配和跟踪）。不幸的是，这些任务忽略了纹理意识，并需要3D数据作为地面真实，这限制了其评估集的规模和多样性。为了解决这些问题，我们引入Feat2GS，从未定位图像中提取的VFM特征中读取3D高斯属性。这使我们能够通过新颖的视图合成探测几何和纹理意识，而无需3D数据。此外，3DGS参数的解耦——几何（x，α，Σ）和纹理（c）——使能够分别分析纹理和几何意识。在Feat2GS下，我们进行了广泛的实验，探测几种VFMs的3D意识，并调查导致3D意识VFM的成分。基于这些发现，我们开发了几种变体，在多样数据集上实现了最先进的性能。这使Feat2GS在探测VFMs中很有用，并作为新颖视图合成的简单而有效的基线。代码和数据可在https://fanegg.github.io/Feat2GS/上获得。
英文摘要：
Given that visual foundation models (VFMs) are trained on extensive datasets but often limited to 2D images, a natural question arises: how well do they understand the 3D world? With the differences in architecture and training protocols (i.e., objectives, proxy tasks), a unified framework to fairly and comprehensively probe their 3D awareness is urgently needed. Existing works on 3D probing suggest single-view 2.5D estimation (e.g., depth and normal) or two-view sparse 2D correspondence (e.g., matching and tracking). Unfortunately, these tasks ignore texture awareness, and require 3D data as ground-truth, which limits the scale and diversity of their evaluation set. To address these issues, we introduce Feat2GS, which readout 3D Gaussians attributes from VFM features extracted from unposed images. This allows us to probe 3D awareness for geometry and texture via novel view synthesis, without requiring 3D data. Additionally, the disentanglement of 3DGS parameters - geometry ($\boldsymbol{x}$, $α$, $Σ$) and texture ($\boldsymbol{c}$) - enables separate analysis of texture and geometry awareness. Under Feat2GS, we conduct extensive experiments to probe the 3D awareness of several VFMs, and investigate the ingredients that lead to a 3D aware VFM. Building on these findings, we develop several variants that achieve state-of-the-art across diverse datasets. This makes Feat2GS useful for probing VFMs, and as a simple-yet-effective baseline for novel-view synthesis. Code and data are available at https://fanegg.github.io/Feat2GS/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
