Current visual foundation models are trained purely on unstructured 2D data, limiting their understanding of 3D structure of objects and scenes. In this work, we show that fine-tuning on 3D-aware data improves the quality of emerging semantic features. We design a method to lift semantic 2D features into an efficient 3D Gaussian representation, which allows us to re-render them for arbitrary views. Using the rendered 3D-aware features, we design a fine-tuning strategy to transfer such 3D awareness into a 2D foundation model. We demonstrate that models fine-tuned in that way produce features that readily improve downstream task performance in semantic segmentation and depth estimation through simple linear probing. Notably, though fined-tuned on a single indoor dataset, the improvement is transferable to a variety of indoor datasets and out-of-domain datasets. We hope our study encourages the community to consider injecting 3D awareness when training 2D foundation models. Project page: https://ywyue.github.io/FiT3D.
Alibaba Group(阿里巴巴集团);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Zhejiang University(浙江大学)
对比图像预训练(CLIP)模型具有zero-shot能力,通过使用图像与提示语句“a [CONTEXT] of [CLASS]"之间的相似性来对属于“[CLASS]”的图像进行分类。基于“[CONTEXT]"中的详尽文本线索,CLIP模型能够感知不同的上下文,例如背景、风格、视点,并对大范围的分布偏移表现出前所未有的鲁棒性。然而,最近的工作发现CLIP模型的进一步微调提高了准确性,但牺牲了下游任务的鲁棒性。我们进行了一项实证调查,以表明微调会破坏预训练CLIP特征的上下文感知能力。为了解决这个问题,我们提出了上下文感知鲁棒微调(CAR-FT)。CAR-FT在微调期间对模型进行规则化,以捕获上下文信息。具体来说,我们使用zero-shot提示权重来获得图像中包含的上下文分布。CAR-FT通过最小化由原始/微调CLIP模型引起的上下文分布之间的Kullback-Leibler分歧(KLD),使CLIP的上下文感知能力继承到下游任务中,并实现更高的分布内(ID)和分布外(OOD)准确性。实验结果表明,CAR-FT在ImageNet的五个OOD测试数据集上具有出色的鲁棒性,同时在九个下游任务上带来了准确率的提高。此外,CAR-FT超越了以前的领域泛化(DG)方法,在DomainBed基准测试中获得了78.5%的平均准确率,建立了新的最先进的水平。
TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model
Institute of Automation, CAS(中国科学院自动化研究所);State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);State Key Lab of Processors(处理器国家重点实验室);Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
SILT: Shadow-aware Iterative Label Tuning for Learning to Detect Shadows from Noisy Labels
The Chinese University of Hong Kong(香港中文大学);The Shun Hing Institute of Advanced Engineering(信兴高等工程研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Confounder-Aware Medical Data Selection for Fine-Tuning Pretrained Vision Models
West China Hospital, Sichuan University(四川大学华西医院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);College of Computer Science, Sichuan University(四川大学计算机学院);School of Biomedical Engineering, University of Science and Technology of China(中国科学技术大学生物医学工程学院)
Bosch Center for AI(博世人工智能中心);University of Tübingen(蒂宾根大学);University of Mannheim(曼海姆大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Bosch Center for Artificial Intelligence(博世人工智能中心)
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Tulane University(路易斯安那州立大学);Carnegie Mellon University(卡内基梅隆大学);Oak Ridge National Laboratory(橡树岭国家实验室);Georgia Institute of Technology(佐治亚理工学院)