arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.18293cs.CV
收藏

视觉-语言模型的高效测试时自适应

Efficient Test-Time Adaptation of Vision-Language Models

Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Nanyang Technological University(南洋理工大学);University of Ottawa(渥太华大学);Carnegie Mellon University(卡内基梅隆大学)

Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, Eric Xing

AI总结:

针对视觉-语言模型测试时自适应计算密集的问题,提出免训练的动态适配器TDA,利用轻量级键值缓存与渐进式伪标签细化实现高效自适应,并引入负伪标签减轻噪声影响,显著提升了有效性与效率。

摘要:

利用预训练的视觉-语言模型进行测试时自适应,在应对测试期间的分布偏移方面引起了越来越多的关注。尽管先前的研究取得了非常出色的性能,但它们涉及密集的计算,这与测试时自适应严重不符。我们设计了TDA,这是一种免训练的动态适配器,能够利用视觉-语言模型实现有效且高效的测试时自适应。TDA与一个轻量级的键值缓存协同工作,该缓存维护一个动态队列,以少样本伪标签作为值,以相应的测试样本特征作为键。利用键值缓存,TDA通过渐进式伪标签细化,允许逐渐适应测试数据,该过程极其高效且不产生任何反向传播。此外,我们引入了负伪标签,当模型对其伪标签预测不确定时,通过将伪标签分配给某些负类,从而减轻伪标签噪声的不利影响。在两个基准上的大量实验表明,与最先进方法相比,TDA具有卓越的有效性和效率。代码已在\url{https://kdiaaa.github.io/tda/}发布。

英文摘要:

Test-time adaptation with pre-trained vision-language models has attracted increasing attention for tackling distribution shifts during the test time. Though prior studies have achieved very promising performance, they involve intensive computation which is severely unaligned with test-time adaptation. We design TDA, a training-free dynamic adapter that enables effective and efficient test-time adaptation with vision-language models. TDA works with a lightweight key-value cache that maintains a dynamic queue with few-shot pseudo labels as values and the corresponding test-sample features as keys. Leveraging the key-value cache, TDA allows adapting to test data gradually via progressive pseudo label refinement which is super-efficient without incurring any backpropagation. In addition, we introduce negative pseudo labeling that alleviates the adverse impact of pseudo label noises by assigning pseudo labels to certain negative classes when the model is uncertain about its pseudo label predictions. Extensive experiments over two benchmarks demonstrate TDA's superior effectiveness and efficiency as compared with the state-of-the-art. The code has been released in \url{https://kdiaaa.github.io/tda/}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑