进步的错觉？对视觉语言模型的测试时适应的批判性审视
The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
AI总结：
针对视觉语言模型测试时适应方法评估不一致的问题，提出统一基准 TTA-VLM，在15个数据集上评估15种方法并引入多维度指标，发现现有方法收益有限且常以牺牲可信度换取准确率。
摘要：
测试时适应（TTA）方法因能在无需额外标注数据的情况下提升 CLIP 等视觉语言模型（VLMs）在推理阶段的性能而备受关注。然而，当前的 TTA 研究普遍存在重大局限，如基线结果重复、评估指标有限、实验设置不一致以及分析不足。这些问题阻碍了 TTA 方法间的公平比较，使其难以评估其实际优缺点。为应对这些挑战，我们引入了 TTA-VLM，一个用于评估 VLMs 上 TTA 方法的综合基准。我们的基准在一个统一且可复现的框架内实现了 8 种情景式 TTA 和 7 种在线 TTA 方法，并在 15 个广泛使用的数据集上进行了评估。与以往仅关注 CLIP 的研究不同，我们将评估扩展至 SigLIP——一种使用 Sigmoid 损失训练的模型——并纳入了 CoOp、MaPLe 和 TeCoA 等训练时调优方法以评估通用性。除分类准确率外，TTA-VLM 还纳入了多种评估指标，包括鲁棒性、校准、分布外检测和稳定性，从而实现对 TTA 方法更全面的评估。通过大量实验，我们发现：1）与以往的开创性工作相比，现有 TTA 方法收益有限；2）当前 TTA 方法与训练时微调方法表现出良好的协作性；3）准确率的提升常以降低模型可信度为代价。我们发布 TTA-VLM 以促进对 VLMs 上 TTA 方法的公平和全面评估，并希望鼓励社区开发可靠且通用的 TTA 策略。
英文摘要：
Test-time adaptation (TTA) methods have gained significant attention for enhancing the performance of vision-language models (VLMs) such as CLIP during inference, without requiring additional labeled data. However, current TTA researches generally suffer from major limitations such as duplication of baseline results, limited evaluation metrics, inconsistent experimental settings, and insufficient analysis. These problems hinder fair comparisons between TTA methods and make it difficult to assess their practical strengths and weaknesses. To address these challenges, we introduce TTA-VLM, a comprehensive benchmark for evaluating TTA methods on VLMs. Our benchmark implements 8 episodic TTA and 7 online TTA methods within a unified and reproducible framework, and evaluates them across 15 widely used datasets. Unlike prior studies focused solely on CLIP, we extend the evaluation to SigLIP--a model trained with a Sigmoid loss--and include training-time tuning methods such as CoOp, MaPLe, and TeCoA to assess generality. Beyond classification accuracy, TTA-VLM incorporates various evaluation metrics, including robustness, calibration, out-of-distribution detection, and stability, enabling a more holistic assessment of TTA methods. Through extensive experiments, we find that 1) existing TTA methods produce limited gains compared to the previous pioneering work; 2) current TTA methods exhibit poor collaboration with training-time fine-tuning methods; 3) accuracy gains frequently come at the cost of reduced model trustworthiness. We release TTA-VLM to provide fair comparison and comprehensive evaluation of TTA methods for VLMs, and we hope it encourages the community to develop more reliable and generalizable TTA strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
