In this paper, we aim to adapt a model at test-time using a few unlabeled data to address distribution shifts. To tackle the challenges of extracting domain knowledge from a limited amount of data, it is crucial to utilize correlated information from pre-trained backbones and source domains. Previous studies fail to utilize recent foundation models with strong out-of-distribution generalization. Additionally, domain-centric designs are not flavored in their works. Furthermore, they employ the process of modelling source domains and the process of learning to adapt independently into disjoint training stages. In this work, we propose an approach on top of the pre-computed features of the foundation model. Specifically, we build a knowledge bank to learn the transferable knowledge from source domains. Conditioned on few-shot target data, we introduce a domain prompt generator to condense the knowledge bank into a domain-specific prompt. The domain prompt then directs the visual features towards a particular domain via a guidance module. Moreover, we propose a domain-aware contrastive loss and employ meta-learning to facilitate domain knowledge extraction. Extensive experiments are conducted to validate the domain knowledge extraction. The proposed method outperforms previous work on 5 large-scale benchmarks including WILDS and DomainNet.
Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Visual Prompting in Multimodal Large Language Models: A Survey
UC San Diego(加州大学圣地亚哥分校);Dartmouth College(达特茅斯学院);Rutgers University(罗格斯大学);UC Los Angeles(加州大学洛杉矶分校);Adobe Research(奥多比研究院);The University of New South Wales(新南威尔士大学);CSIRO’s Data61(联邦科学与工业研究组织Data61)
GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing
School of Computing, Kyung Hee University(京畿大学计算机学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);College of Computer Science and Information Engineering, Harbin Normal University(哈尔滨师范大学计算机科学与信息工程学院);College of Professional and Continuing Education, The Hong Kong Polytechnic University(香港理工大学专业及继续教育学院);School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院)
Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrieval
College of Computer Science, Sichuan University(四川大学计算机学院);Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu, China(教育部机器学习与产业智能工程研究中心);Deep NeuroCognition Lab, I2R and CFAR, Agency for Science, Technology and Research, Singapore(深度神经认知实验室,I2R和CFAR,科技研究局,新加坡)
Adapting Vision Foundation Models with Cascaded Semantics
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Carnegie Mellon University(卡内基梅隆大学);University of Missouri–Kansas City(密苏里大学堪萨斯城分校);Northeastern University(东北大学);Tulane University(杜兰大学);University of Bristol(布里斯托大学);Oak Ridge National Laboratory(橡树岭国家实验室)
People, Penguins and Petri Dishes: Adapting Object Counting Models To New Visual Domains And Object Types Without Forgetting
在本文中,我们提出了一种技术,以适应卷积神经网络(CNN)为基础的对象计数器,以额外的视觉领域和对象类型,同时仍然保留原来的计数功能。训练特定于域的归一化和缩放运算符,以允许模型根据各种视觉域的统计分布进行调整。开发的适应技术是用来产生一个单一的补丁为基础的计数回归能够计数各种对象类型,包括人,车辆,细胞核和野生动物。作为本研究的一部分,构建了组织培养和患者诊断背景下具有挑战性的新细胞计数数据集。这个新的集合,被称为都柏林细胞计数(DCC)数据集,是第一个提供给更广泛的计算机视觉社区。在Shanghaitech(A和B部分)和Penguins数据集中实现了最先进的对象计数性能,而在TRANCOS和Modified Bone Marrow(MBM)数据集上观察到了竞争性性能,所有数据集都使用共享计数模型。
Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking
University of Washington(华盛顿大学);Electronics and Telecommunications Research Institute(电子电信研究所);National Center for High-performance Computing(高性能计算国家中心)
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别新实验室)
CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding
Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Peng Cheng Laboratory (PCL)(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院)
Exploring Interpretability for Visual Prompt Tuning with Cross-layer Concepts
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Microsoft Research Asia(微软亚洲研究院);School of Telecommunication and Engineering, Xidian University(西安电子科技大学电信与工程学院)
Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室)
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));vivo Mobile Communication Co., Ltd.(维沃移动通信有限公司);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Pengcheng Laboratory(鹏城实验室)
MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University(国家多媒体软件工程研究中心,计算机学院,武汉大学);Hong Kong University of Science and Technology(香港科学与技术大学);Horizon Robotics