arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.03729cs.CV
收藏

视觉-语言模型的现实测试时自适应

Realistic Test-Time Adaptation of Vision-Language Models

UCLouvain(天主教鲁汶大学);UMons(蒙斯大学);ÉTS Montreal(蒙特利尔工程学院)

Maxime Zanella, Clément Fuchs, Christophe De Vleeschouwer, Ismail Ben Ayed

AI总结:

本文指出现有VLM测试时自适应方法依赖全类别等不现实假设,提出含可变有效类别和非i.i.d.批次的评估框架,并推出以统计锚点正则保护文本编码器知识的StatA方法。

摘要:

视觉-语言模型(VLM)的零样本能力已被广泛用于提升预测性能。然而,以往关于直推式或测试时自适应(TTA)的工作往往对数据分布作出强假设,例如所有类别都会出现。我们的工作对这些有利的部署场景提出质疑,并引入了一个更现实的评估框架,包括:(i)单个批次内用于自适应的有效类别数量可变;(ii)在线自适应设置中的测试样本为非独立同分布批次。我们提供了全面的评估、比较和消融研究,证明当前用于VLM的直推式或TTA方法如何在各种现实场景中系统性地损害模型最初的零样本鲁棒性,而偏向于在关于测试样本分布的有利假设下取得性能提升。此外,我们提出了StatA,这是一种能够处理广泛部署场景的通用方法,包括测试时有效类别数量可变的场景。我们的方法加入了一个专为VLM设计的新颖正则化项,其作用相当于统计锚点,可保留初始的文本编码器知识,尤其是在低数据条件下。代码可在 https://github.com/MaxZanella/StatA 获取。

英文摘要:

The zero-shot capabilities of Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance. However, previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution, such as the presence of all classes. Our work challenges these favorable deployment scenarios, and introduces a more realistic evaluation framework, including: (i) a variable number of effective classes for adaptation within a single batch, and (ii) non-i.i.d. batches of test samples in online adaptation settings. We provide comprehensive evaluations, comparisons, and ablation studies that demonstrate how current transductive or TTA methods for VLMs systematically compromise the models' initial zero-shot robustness across various realistic scenarios, favoring performance gains under advantageous assumptions about the test samples' distributions. Furthermore, we introduce StatA, a versatile method that could handle a wide range of deployment scenarios, including those with a variable number of effective classes at test time. Our approach incorporates a novel regularization term designed specifically for VLMs, which acts as a statistical anchor preserving the initial text-encoder knowledge, particularly in low-data regimes. Code available at https://github.com/MaxZanella/StatA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑