arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.18330cs.CVcs.AI
收藏

视觉-语言模型的极其简单的测试时自适应方法

Frustratingly Easy Test-Time Adaptation of Vision-Language Models

University of Trento(特伦托大学);U2IS, ENSTA Paris, Institut Polytechnique de Paris(巴黎综合理工学院 ENSTA 巴黎分校 U2IS);Fondazione Bruno Kessler (FBK)(布鲁诺·凯塞勒基金会)

Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, Elisa Ricci

AI总结:

针对视觉-语言模型在挑战性样本上泛化差的问题,本文提出了名为ZERO的简单测试时自适应方法,仅需单次前向传播,性能比肩或超越现有最优方法,速度快10倍、内存效率高13倍,可作为领域强基线。

摘要:

视觉-语言模型(Vision-Language Models, VLMs)能够对任意语义类别进行无缝判别,但在面对具有挑战性的样本时,其泛化能力仍然较差。为此,情景式测试时自适应(Episodic Test-Time Adaptation, TTA)策略近期成为一类强大的技术,可在仅提供单张无标注图像的情况下对VLM进行自适应调整。近期的TTA研究主流是基于边际熵最小化的提示调优范式,该方法依赖在线反向传播,不可避免地会降低推理速度,同时增加内存占用。在本研究中,我们从理论层面分析了该方法的特性,并揭示出一种效果惊人的TTA方法一直潜藏其中。我们将该方法命名为ZERO(即“零”温度的TTA),其设计不仅效果极佳,而且简单得令人意外:对图像进行N次增强,执行预测,保留置信度最高的预测结果,并在将Softmax温度设为零后进行边际化处理。值得注意的是,ZERO仅需要对视觉编码器执行单次批量前向传播,无需任何反向传播。我们遵循文献中确立的实验协议对所提方法进行了全面评估,结果表明ZERO的性能大幅超越现有最优方法或与之相当,同时其速度比标准测试时提示调愉快近10倍,内存效率高出13倍。凭借其简单性和可忽略不计的计算开销,ZERO可作为该领域未来研究的强基线。代码可在https://github.com/FarinaMatteo/zero获取。

英文摘要:

Vision-Language Models seamlessly discriminate among arbitrary semantic categories, yet they still suffer from poor generalization when presented with challenging examples. For this reason, Episodic Test-Time Adaptation (TTA) strategies have recently emerged as powerful techniques to adapt VLMs in the presence of a single unlabeled image. The recent literature on TTA is dominated by the paradigm of prompt tuning by Marginal Entropy Minimization, which, relying on online backpropagation, inevitably slows down inference while increasing memory. In this work, we theoretically investigate the properties of this approach and unveil that a surprisingly strong TTA method lies dormant and hidden within it. We term this approach ZERO (TTA with "zero" temperature), whose design is both incredibly effective and frustratingly simple: augment N times, predict, retain the most confident predictions, and marginalize after setting the Softmax temperature to zero. Remarkably, ZERO requires a single batched forward pass through the vision encoder only and no backward passes. We thoroughly evaluate our approach following the experimental protocol established in the literature and show that ZERO largely surpasses or compares favorably w.r.t. the state-of-the-art while being almost 10x faster and 13x more memory-friendly than standard Test-Time Prompt Tuning. Thanks to its simplicity and comparatively negligible computation, ZERO can serve as a strong baseline for future work in this field. The code is available at https://github.com/FarinaMatteo/zero.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑