少即是多？探索 Token 凝缩作为免训练测试时适应方法
Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation
AI总结：
针对 CLIP 等 VLM 零样本跨域性能下降问题，论文提出免训练的 TCA，通过域锚定 token 凝缩与 logits 校正提升适应效果并降低计算量。
摘要：
对比语言-图像预训练（CLIP）擅长学习可泛化的图像表征，但在某些下游数据集上进行零样本推理时往往表现不佳。测试时适应（TTA）通过调整归一化层或上下文提示等组件来缓解这一问题，然而它通常需要较大的批量规模和大量增强操作，导致高昂的计算成本。这引出了一个关键问题：视觉语言模型（VLM）在特定测试场景中的性能下降，能否通过高效且免训练的方法得到缓解？为探索解决方案，作者研究了 token 凝缩（TC）技术；该技术最初旨在通过在推理过程中优化 token 使用来提升 vision transformer 的效率。作者观察到，富含信息的 token 能够改善 CLIP 等 VLM 在未见过数据集上的视觉-文本对齐。然而，现有 TC 方法在减少 token 时往往无法维持分布内性能，这促使作者追问：如何将 TC 转化为 VLM 有效的“免费午餐”适应策略？为此，论文提出 Token Condensation as Adaptation（TCA），这是一种超越标准 TC 的免训练适应方法。TCA 并非被动丢弃 token，而是通过引入基于储备池的域锚定 token 来凝缩 token 表征，从而实现保留信息的 token 约简和 logits 校正。在跨数据集基准和 CIFAR-100-Corrupted 数据集上，TCA 相比最强基线取得最高 21.4% 的性能提升，同时将 GFLOPs 降低 12.2% 至 48.9%，并且在 CLIP 和 SigLIP 系列上对超参数的依赖都很小。
英文摘要：
Contrastive Language-Image Pretraining (CLIP) excels at learning generalizable image representations but often falls short in zero-shot inference on certain downstream datasets. Test-time adaptation (TTA) mitigates this issue by adjusting components like normalization layers or context prompts, yet it typically requires large batch sizes and extensive augmentations, leading to high computational costs. This raises a key question: Can VLMs' performance drop in specific test cases be mitigated through efficient, training-free approaches? To explore the solution, we investigate token condensation (TC) techniques, originally designed to enhance vision transformer efficiency by refining token usage during inference. We observe that informative tokens improve visual-text alignment in VLMs like CLIP on unseen datasets. However, existing TC methods often fail to maintain in-distribution performance when reducing tokens, prompting us to ask: How can we transform TC into an effective ``free-lunch'' adaptation strategy for VLMs? To address this, we propose Token Condensation as Adaptation (TCA), a training-free adaptation method that takes a step beyond standard TC. Rather than passively discarding tokens, TCA condenses token representation by introducing reservoir-based domain anchor tokens for information-preserving token reduction and logits correction. TCA achieves up to a 21.4% performance improvement over the strongest baseline on cross-dataset benchmark and the CIFAR-100-Corrupted dataset while reducing GFLOPs by 12.2% to 48.9%, with minimal hyperparameter dependency on both CLIP and SigLIP series.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
