大型视觉-语言适配器的稳健校准
Robust Calibration of Large Vision-Language Adapters
AI总结：
针对CLIP适配中OOD场景下的校准误差问题，提出通过缩放logit范围至零样本预测的简单模型无关方法，在保持判别性能的同时有效提升校准能力。
摘要：
本文解决了基于CLIP的模型适配中校准误差这一关键问题，特别是在分布外（OOD）样本这一具有挑战性的场景下，而现有关于CLIP适配的文献对此关注不足。我们通过实验证明，流行的CLIP适配方法，如适配器（Adapters）、提示学习（Prompt Learning）和测试时适配（Test-Time Adaptation），在存在分布漂移时，会显著降低零样本基线的校准能力。我们识别出logit范围的增大是导致CLIP适配方法校准误差的根本原因，这与先前针对全监督模型校准的研究形成对比。基于这些观察，我们提出了一种简单且与模型无关的解决方案来缓解校准误差，即通过将每个样本的logit范围缩放到其零样本预测的logit。我们探索了三种不同的实现方式，这些方式可以在适配过程中集成，也可以直接在推理时使用。在流行的OOD分类基准上的综合实验表明，所提出的方法在缓解校准误差的同时保持了判别性能，其改进在日益流行的三类方法中保持一致。代码公开于：https://github.com/Bala93/CLIPCalib
英文摘要：
This paper addresses the critical issue of miscalibration in CLIP-based model adaptation, particularly in the challenging scenario of out-of-distribution (OOD) samples, which has been overlooked in the existing literature on CLIP adaptation. We empirically demonstrate that popular CLIP adaptation approaches, such as Adapters, Prompt Learning, and Test-Time Adaptation, substantially degrade the calibration capabilities of the zero-shot baseline in the presence of distributional drift. We identify the increase in logit ranges as the underlying cause of miscalibration of CLIP adaptation methods, contrasting with previous work on calibrating fully-supervised models. Motivated by these observations, we present a simple and model-agnostic solution to mitigate miscalibration, by scaling the logit range of each sample to its zero-shot prediction logits. We explore three different alternatives to achieve this, which can be either integrated during adaptation or directly used at inference time. Comprehensive experiments on popular OOD classification benchmarks demonstrate the effectiveness of the proposed approaches in mitigating miscalibration while maintaining discriminative performance, whose improvements are consistent across the three families of these increasingly popular approaches. The code is publicly available at: https://github.com/Bala93/CLIPCalib
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
