发表机构
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences; Institute of Automation, Chinese Academy of Sciences; School of Artificial Intelligence, University of Chinese Academy of Sciences; Nanjing University（中国科学院大学先进交叉科学学院; 中国科学院自动化研究所; 中国科学院大学人工智能学院; 南京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对测试时适应中的无效或有害适应问题，提出选择性适应方法CAS，通过跨增强相似度判断是否跳过适应，在跳过85%过程时仍保持或提升准确率。
AI 中文摘要
测试时适应（TTA）已成为在推理过程中使视觉语言模型适应分布偏移的一种重要策略。我们对适应前后的模型预测进行了逐样本分析，并观察到现有TTA方法中存在两种与先前工作相呼应的失败模式。适应经常是微不足道的，对模型的预测没有产生任何改变；更严重的是，适应可能是有害的，将原本正确的预测翻转为错误的预测。这自然引出一个问题：我们能否识别并跳过这些微不足道或有害的适应？在这项工作中，我们引入了选择性适应这一新问题，旨在确定给定的测试样本应该进行适应还是被跳过。为此，我们提出了跨增强相似度（CAS），这是一个简单的基线方法，仅当增强视图之间的预测表现出低相似度时才执行适应。值得注意的是，即使跳过了近85%的适应过程，CAS不仅保持了总体准确率，在某些情况下还提高了准确率。我们希望其他研究者能够探索这一新方向，并超越我们基线的性能。我们的代码可在以下网址获取：https URL。
英文摘要
Test-time adaptation (TTA) has emerged as a prominent strategy for adapting vision-language models to distribution shifts during inference. We conduct a per-sample analysis of model predictions before and after adaptation, and observe two failure modes in existing TTA methods that echo previous work. Adaptations are frequently negligible, yielding no change in the model's predictions, and more severely, they can be detrimental by flipping previously correct predictions to incorrect ones. This naturally raises a question: Can we identify and skip such negligible or harmful adaptations? In this work, we introduce a new problem of selective adaptation, which aims to determine whether a given test sample should undergo adaptation or be skipped. To this end, we propose Cross-Augmentation Similarity (CAS), a simple baseline that performs adaptation only when predictions across augmented views exhibit low similarity. Notably, CAS not only preserves but in some cases improves overall accuracy, even when skipping nearly 85% of the adaptation process. We hope other researchers will explore this new direction and surpass the performance of our baseline. Our code is available at https://github.com/sirujiang/selective-adaptation.
CommentsECCV 2026
