The zero-shot capabilities of Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance. However, previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution, such as the presence of all classes. Our work challenges these favorable deployment scenarios, and introduces a more realistic evaluation framework, including: (i) a variable number of effective classes for adaptation within a single batch, and (ii) non-i.i.d. batches of test samples in online adaptation settings. We provide comprehensive evaluations, comparisons, and ablation studies that demonstrate how current transductive or TTA methods for VLMs systematically compromise the models' initial zero-shot robustness across various realistic scenarios, favoring performance gains under advantageous assumptions about the test samples' distributions. Furthermore, we introduce StatA, a versatile method that could handle a wide range of deployment scenarios, including those with a variable number of effective classes at test time. Our approach incorporates a novel regularization term designed specifically for VLMs, which acts as a statistical anchor preserving the initial text-encoder knowledge, particularly in low-data regimes. Code available at https://github.com/MaxZanella/StatA.
RADC: Risk-Aware Dual Caching for Vision-Language Test-Time Adaptation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Pengcheng Laboratory(鹏城实验室);School of Intelligent Systems Engineering, Sun Yat-sen University(中山大学智能工程学院)
Efficient Test-Time Adaptation of Vision-Language Models
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Nanyang Technological University(南洋理工大学);University of Ottawa(渥太华大学);Carnegie Mellon University(卡内基梅隆大学)
MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptation
China University of Petroleum-Beijing at Karamay(中国石油大学(北京)克拉玛依校区);Guizhou University(贵州大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);Shanghai Jiao Tong University(上海交通大学)
Flatness Guided Test-Time Adaptation for Vision-Language Models
School of Cyber Science and Technology, University of Science and Technology of China(中国科学技术大学网络科学与技术学院);National Engineering Laboratory for Brain-Inspired Intelligence Technology and Applications, University of Science and Technology of China(中国科学技术大学脑启发式智能技术与应用国家工程实验室);Peng Cheng Laboratory, Shenzhen, China(深圳鹏城实验室)
Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model
South China University of Technology(华南理工大学);School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Shanghai AI Laboratory(上海人工智能实验室)
Koo-Fu CLIP: Closed-Form Adaptation of Vision-Language Models via Fukunaga-Koontz Linear Discriminant Analysis
Visual Recognition Group(视觉识别组);Department of Cybernetics(cybernetics系);Faculty of Electrical Engineering(电气工程系);Czech Technical University in Prague(布拉格捷克技术大学)
Scaling Down to Scale Up: Towards Operationally-Efficient and Deployable Clinical Models via Cross-Modal Low-Rank Adaptation for Medical Vision-Language Models
King Fahd University of Petroleum(国王法赫德石油与矿物大学);Institute for Medical Engineering(医学工程研究所);Science, Massachusetts Institute of Technology, US(科学,麻省理工学院,美国);Harvard Medical School, Harvard University, US(哈佛医学院,哈佛大学,美国);SDAIA-KFUPM Joint Research Center for Artificial Intelligence, Saudi Arabia(SDAIA-KFUPM人工智能联合研究中心,沙特阿拉伯)
Local Margin Restoration for Test-Time Adaptation of Vision-Language Models
Guangzhou University(广州大学);The Second Affiliated Hospital of Guangzhou University of Chinese Medicine(广州中医药大学第二附属医院);Jinan University(暨南大学);Pengcheng Laboratory(鹏城实验室)
To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing University(南京大学)
Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation
Institute of Automation, CAS(中国科学院自动化研究所);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究所);Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
Training-Free Test-Time Adaptation with Brownian Distance Covariance in Vision-Language Models
College of Computer Science and Software Engineering, Shenzhen University, China(深圳大学计算机科学与软件工程学院);University of Cambridge, Cambridge, UK(剑桥大学);Yau Mathematical Sciences Center, Tsinghua University, Beijing, China(清华大学尤里伊数学科学中心);Southern University of Science and Technology, Shenzhen, China(南方科技大学)
Guided Prompt Evolution for Vision-Language Models Adaptation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);Sun Yat-sen University(中山大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))