arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.05787cs.CV
收藏

特征上采样器在2D到3D场景重建中的频谱探测

Spectral Probing of Feature Upsamplers in 2D-to-3D Scene Reconstruction

Hokkaido University(北海道大学);Westlake University(西湖大学);Zhejiang University(浙江大学);The University of Tokyo(东京大学)

Ling Xiao, Yuliang Xiu, Yue Chen, Guoming Wang, Toshihiko Yamasaki

AI总结:

本文提出频谱诊断框架,通过六个指标评估特征上采样器对3D重建的影响,发现频谱一致性比空间细节更关键。

摘要:

典型的2D到3D流程以多视角图像作为输入,其中视觉基础模型(VFM)提取的特征被空间上采样为密集表示,用于3D重建。如果不同视角的密集特征保持几何一致性,可微渲染可以恢复准确的3D表示,使特征上采样器成为关键组件。最近可学习的上采样方法主要旨在增强空间细节,例如更清晰的几何或更丰富的纹理,但它们对3D意识的影响仍不明确。为了解决这一差距,我们引入了一个具有六个互补指标的频谱诊断框架,这些指标表征幅度再分布、结构频谱对齐和方向稳定性。在CLIP和DINO后端的经典插值和可学习上采样方法上,我们观察到三个关键发现。第一,结构频谱一致性(SSC/CSC)是NVS质量最强的预测因子,而高频频谱斜率漂移(HFSS)往往与重建性能呈负相关,表明强调高频细节并不一定改善3D重建。第二,几何和纹理对不同的频谱特性作出反应:角能量一致性(ADC)与几何相关指标相关性更强,而SSC/CSC对纹理保真度的影响略大于几何准确性。第三,尽管可学习的上采样器通常产生更清晰的空间特征,但它们很少在重建质量上超过经典插值,其效果取决于重建模型。总体而言,我们的结果表明,重建质量更接近于保持频谱结构,而不是增强空间细节,突显频谱一致性作为设计2D到3D流程中上采样策略的重要原则。

英文摘要:

A typical 2D-to-3D pipeline takes multi-view images as input, where a Vision Foundation Model (VFM) extracts features that are spatially upsampled to dense representations for 3D reconstruction. If dense features across views preserve geometric consistency, differentiable rendering can recover an accurate 3D representation, making the feature upsampler a critical component. Recent learnable upsampling methods mainly aim to enhance spatial details, such as sharper geometry or richer textures, yet their impact on 3D awareness remains underexplored. To address this gap, we introduce a spectral diagnostic framework with six complementary metrics that characterize amplitude redistribution, structural spectral alignment, and directional stability. Across classical interpolation and learnable upsampling methods on CLIP and DINO backbones, we observe three key findings. First, structural spectral consistency (SSC/CSC) is the strongest predictor of NVS quality, whereas High-Frequency Spectral Slope Drift (HFSS) often correlates negatively with reconstruction performance, indicating that emphasizing high-frequency details alone does not necessarily improve 3D reconstruction. Second, geometry and texture respond to different spectral properties: Angular Energy Consistency (ADC) correlates more strongly with geometry-related metrics, while SSC/CSC influence texture fidelity slightly more than geometric accuracy. Third, although learnable upsamplers often produce sharper spatial features, they rarely outperform classical interpolation in reconstruction quality, and their effectiveness depends on the reconstruction model. Overall, our results indicate that reconstruction quality is more closely related to preserving spectral structure than to enhancing spatial detail, highlighting spectral consistency as an important principle for designing upsampling strategies in 2D-to-3D pipelines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑