arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.20229cs.CV
收藏

通过3D感知微调改进2D特征表示

Improving 2D Feature Representations by 3D-Aware Fine-Tuning

ETH Zurich(苏黎世联邦理工学院);Google(谷歌公司);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所)

Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, Jan Eric Lenssen

AI总结:

本文提出一种3D感知微调方法,将2D语义特征提升为3D高斯表示并重新渲染,以将3D感知能力注入2D基础模型,从而显著提升语义分割和深度估计等下游任务在多种数据集上的性能。

摘要:

当前的视觉基础模型纯粹在非结构化的2D数据上训练,限制了它们对物体和场景3D结构的理解。在这项工作中,我们展示了在3D感知数据上进行微调可以提升涌现的语义特征的质量。我们设计了一种方法,将语义2D特征提升为高效的3D高斯表示,从而允许我们为任意视角重新渲染它们。利用渲染的3D感知特征,我们设计了一种微调策略,将这种3D感知能力迁移到2D基础模型中。我们证明,以这种方式微调的模型所产生的特征,通过简单的线性探测,能够直接提升语义分割和深度估计等下游任务的性能。值得注意的是,尽管仅在单一室内数据集上进行微调,这种改进仍可迁移到各种室内数据集和域外数据集。我们希望这项研究能鼓励社区在训练2D基础模型时考虑注入3D感知。项目页面:https://ywyue.github.io/FiT3D。

英文摘要:

Current visual foundation models are trained purely on unstructured 2D data, limiting their understanding of 3D structure of objects and scenes. In this work, we show that fine-tuning on 3D-aware data improves the quality of emerging semantic features. We design a method to lift semantic 2D features into an efficient 3D Gaussian representation, which allows us to re-render them for arbitrary views. Using the rendered 3D-aware features, we design a fine-tuning strategy to transfer such 3D awareness into a 2D foundation model. We demonstrate that models fine-tuned in that way produce features that readily improve downstream task performance in semantic segmentation and depth estimation through simple linear probing. Notably, though fined-tuned on a single indoor dataset, the improvement is transferable to a variety of indoor datasets and out-of-domain datasets. We hope our study encourages the community to consider injecting 3D awareness when training 2D foundation models. Project page: https://ywyue.github.io/FiT3D.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑