用于微调CLIP的移位感知校准：利用图像-文本对齐
Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment
AI总结：
研究针对微调CLIP时预测置信度与准确性不一致问题，提出无需训练的移位感知校准（SAC）方法，利用原始与微调CLIP输出对数差异作校准信号，经实验验证该方法在多数据集和微调方法上提升了校准效果。
摘要：
视觉语言模型（如CLIP）通过提示调整有效地适应下游任务，但微调可能会使预测置信度和准确性不一致，特别是在未见类别上。现有的特定于VLM的校准方法主要依赖于训练类别的文本特征，限制了它们在类别和分布变化中的适用性。我们提出了移位感知校准（SAC），这是一种无需训练的逐样本方法，它使用原始CLIP和微调CLIP的输出对数之间的差异作为校准信号。我们明确将此量称为“对数移位”，因为它是在预测空间中测量的，而不是在隐藏表示之间。SAC将此对数移位映射到一个正缩放因子，该因子在保持预测类别的同时调整置信度。在11个数据集和五种微调方法上，SAC提高了训练和未见类别的校准，并在跨数据集和域泛化评估下也有提升。互信息（MI）分析表明，对数移位信号与跨方法和数据集的预期校准误差（ECE）之间存在统计依赖性，为SAC使用的信号对校准具有信息性提供了经验证据，并支持其移位感知设计的有效性。
英文摘要：
Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes. Existing VLM-specific calibration methods mainly rely on textual features of train classes, limiting their applicability across class and distribution shifts. We propose \textbf{Shift-Aware Calibration (SAC)}, a training-free, sample-wise method that uses the discrepancy between the output logits of the original and fine-tuned CLIP as a calibration signal. We explicitly call this quantity \emph{logit shift}, since it is measured in prediction space rather than between hidden representations. SAC maps this logit shift to a positive scaling factor that adjusts confidence while preserving the predicted class. Across 11 datasets and five fine-tuning methods, SAC improves calibration on train and unseen classes and under cross-dataset and domain-generalization evaluation. Mutual Information (MI) analysis shows a statistical dependency between the logit-shift signal and ECE across methods and datasets, providing empirical evidence that the signal used by SAC is informative for calibration and supporting the effectiveness of its shift-aware design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
