arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06032cs.CV

Splat和Distill:通过馈送式3D重建增强教师模型以实现3D感知的蒸馏

Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillation

Department of Computer Science(计算机科学系);The Hebrew University of Jerusalem(耶路撒冷希伯来大学)

David Shavin, Sagie Benaim

AI总结:

Splat和Distill通过馈送式3D重建增强教师模型,提升2D VFMs的3D意识和语义丰富性。

摘要:

视觉基础模型(VFMs)在应用于各种下游2D任务时取得了显著的成功。尽管其有效性,它们往往表现出关键的3D意识缺乏。为此,我们引入了Splat和Distill,一个框架,通过在教师模型中添加一个快速的馈送式3D重建流程,将鲁棒的3D意识注入到2D VFMs中。给定由教师模型产生的2D特征,我们的方法首先将这些特征提升为显式的3D高斯表示,以一种前馈的方式。这些3D特征然后被“投射”到新的视角上,产生一组新的2D特征图,用于监督学生模型,从而“蒸馏”出几何上扎根的知识。通过用我们的前馈提升方法替代先前工作的慢速每场景优化,我们的框架避免了特征平均的伪影,创造了一个动态的学习过程,其中教师的一致性随着学生的一致性而提高。我们在一系列下游任务上进行了全面的评估,包括单目深度估计、表面法线估计、多视角对应和语义分割。我们的方法显著优于先前的工作,不仅在3D意识方面取得了显著的提升,还增强了底层的语义丰富性。项目页面可在https://davidshavin4.github.io/Splat-and-Distill/上找到。

英文摘要:

Vision Foundation Models (VFMs) have achieved remarkable success when applied to various downstream 2D tasks. Despite their effectiveness, they often exhibit a critical lack of 3D awareness. To this end, we introduce Splat and Distill, a framework that instills robust 3D awareness into 2D VFMs by augmenting the teacher model with a fast, feed-forward 3D reconstruction pipeline. Given 2D features produced by a teacher model, our method first lifts these features into an explicit 3D Gaussian representation, in a feedforward manner. These 3D features are then ``splatted" onto novel viewpoints, producing a set of novel 2D feature maps used to supervise the student model, ``distilling" geometrically grounded knowledge. By replacing slow per-scene optimization of prior work with our feed-forward lifting approach, our framework avoids feature-averaging artifacts, creating a dynamic learning process where the teacher's consistency improves alongside that of the student. We conduct a comprehensive evaluation on a suite of downstream tasks, including monocular depth estimation, surface normal estimation, multi-view correspondence, and semantic segmentation. Our method significantly outperforms prior works, not only achieving substantial gains in 3D awareness but also enhancing the underlying semantic richness of 2D features. Project page is available at https://davidshavin4.github.io/Splat-and-Distill/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑