微调如何影响视觉语言模型的分布外检测？
How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
AI总结：
本文全面研究微调对视觉语言模型少样本下游任务分布外检测的影响，发现合理选择OOD分数至关重要，且最大概念匹配分数与提示学习结合可实现最优检测性能。
摘要：
近期如CLIP等大型视觉语言模型在分布外（OOD）检测和泛化方面表现出色。然而，它们在下游数据集上的零样本分布内（ID）准确率往往有限。近期基于CLIP的微调方法（如提示学习）在有OOD标签的情况下，在ID分类和OOD泛化方面取得了显著提升。尽管如此，在没有OOD标签的情况下，模型对语义偏移是否可靠仍不清楚。本文旨在填补这一空白，全面研究微调如何影响少样本下游任务的OOD检测。通过将OOD检测构建为多模态概念匹配，我们建立了微调方法与各种OOD分数之间的联系。结果表明，合理选择OOD分数对基于CLIP的微调至关重要。特别是，最大概念匹配（MCM）分数始终提供了一种有前景的解决方案。我们还证明，提示学习在OOD检测上实现了优于零样本基线的最先进性能。
英文摘要：
Recent large vision-language models such as CLIP have shown remarkable out-of-distribution (OOD) detection and generalization performance. However, their zero-shot in-distribution (ID) accuracy is often limited for downstream datasets. Recent CLIP-based fine-tuning methods such as prompt learning have demonstrated significant improvements in ID classification and OOD generalization where OOD labels are available. Nonetheless, it remains unclear whether the model is reliable to semantic shifts without OOD labels. In this paper, we aim to bridge the gap and present a comprehensive study to understand how fine-tuning impact OOD detection for few-shot downstream tasks. By framing OOD detection as multi-modal concept matching, we establish a connection between fine-tuning methods and various OOD scores. Our results suggest that a proper choice of OOD scores is essential for CLIP-based fine-tuning. In particular, the maximum concept matching (MCM) score provides a promising solution consistently. We also show that prompt learning demonstrates the state-of-the-art OOD detection performance over the zero-shot counterpart.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
