几何遇见视觉：重新审视蒸馏场中的预训练语义
Geometry Meets Vision: Revisiting Pretrained Semantics in Distilled Fields
AI总结：
本文评估几何接地特征在蒸馏辐射场中的作用，并提出无需初始位姿的 SPINE 反演框架，发现几何细节更丰富却会降低位姿精度，纯视觉特征通用性更强。
摘要：
辐射场中的语义蒸馏推动了开放词汇机器人策略的显著进步，例如在操作和导航中，这些策略建立在来自大型视觉模型的预训练语义之上。尽管先前工作已经证明了纯视觉语义特征（例如 DINO 和 CLIP）在 Gaussian Splatting 和神经辐射场中的有效性，但几何接地在蒸馏场中的潜在益处仍是一个悬而未决的问题。原则上，视觉-几何特征对于位姿估计等空间任务似乎非常有前景，这引出了一个问题：几何接地的语义特征在蒸馏场中是否具有优势？具体而言，我们提出三个关键问题：第一，空间接地是否会产生更高保真度、具备几何感知能力的语义特征？我们发现，与对应特征相比，来自几何接地骨干网络的图像特征包含更精细的结构细节。第二，几何接地是否改善语义目标定位？我们观察到该任务没有显著差异。第三，几何接地是否能够实现更高精度的辐射场反演？鉴于先前工作的局限及其缺乏语义整合，我们提出了一个无需初始猜测即可反演辐射场的新颖框架 SPINE，它由两个核心组件组成：使用蒸馏语义的粗略反演，以及使用基于光度优化的精细反演。令人惊讶的是，我们发现使用几何接地特征时位姿估计精度反而下降。我们的结果表明，尽管几何接地特征包含更多几何细节，但纯视觉特征在更广泛的下游任务中具有更强的通用性。值得注意的是，我们的发现强调了未来有必要研究有效的几何接地策略，以增强预训练语义特征的通用性和性能。
英文摘要：
Semantic distillation in radiance fields has spurred significant advances in open-vocabulary robot policies, e.g., in manipulation and navigation, founded on pretrained semantics from large vision models. While prior work has demonstrated the effectiveness of visual-only semantic features (e.g., DINO and CLIP) in Gaussian Splatting and neural radiance fields, the potential benefit of geometry-grounding in distilled fields remains an open question. In principle, visual-geometry features seem very promising for spatial tasks such as pose estimation, prompting the question: Do geometry-grounded semantic features offer an edge in distilled fields? Specifically, we ask three critical questions: First, does spatial-grounding produce higher-fidelity geometry-aware semantic features? We find that image features from geometry-grounded backbones contain finer structural details compared to their counterparts. Secondly, does geometry-grounding improve semantic object localization? We observe no significant difference in this task. Thirdly, does geometry-grounding enable higher-accuracy radiance field inversion? Given the limitations of prior work and their lack of semantics integration, we propose a novel framework SPINE for inverting radiance fields without an initial guess, consisting of two core components: coarse inversion using distilled semantics, and fine inversion using photometric-based optimization. Surprisingly, we find that the pose estimation accuracy decreases with geometry-grounded features. Our results suggest that visual-only features offer greater versatility for a broader range of downstream tasks, although geometry-grounded features contain more geometric detail. Notably, our findings underscore the necessity of future research on effective strategies for geometry-grounding that augment the versatility and performance of pretrained semantic features.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
