弥合模态差距：基于多模态原型和图像偏差估计的少样本分布外检测
Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation
AI总结：
本文提出SUPREME框架，通过引入多模态原型和图像偏差估计，有效缓解图像与文本之间的模态差距，提升少样本分布外检测性能。
摘要：
现有的基于视觉-语言模型（VLM）的分布外（OOD）检测方法通常依赖于输入图像与分布内（ID）文本原型之间的相似度分数。然而，图像与文本之间的模态差距往往导致较高的误报率，因为OOD样本可能与ID文本原型表现出高度相似性。为减轻这种模态差距的影响，我们提出引入ID图像原型和ID文本原型。我们进行了理论分析和实证研究，表明这种方法在不增加额外训练的情况下可以提升VLM基于的OOD检测性能。为进一步减少图像与文本之间的差距，我们引入了一个新的少样本调优框架SUPREME，包括偏置提示生成（BPG）和图像-文本一致性（ITC）模块。BPG通过将ID文本原型条件在基于高斯估计的图像领域偏置上，增强图像-文本融合并提高泛化能力；ITC通过最小化内模和跨模距离来减少模态差距。此外，受我们的理论和实证发现的启发，我们引入了一个新的OOD分数S_GMP，利用单模和跨模相似性。最后，我们进行了广泛的实验，证明SUPREME在一致性上优于现有的VLM基于的OOD检测方法。
英文摘要：
Existing vision-language model (VLM)-based methods for out-of-distribution (OOD) detection typically rely on similarity scores between input images and in-distribution (ID) text prototypes. However, the modality gap between image and text often results in high false positive rates, as OOD samples can exhibit high similarity to ID text prototypes. To mitigate the impact of this modality gap, we propose incorporating ID image prototypes along with ID text prototypes. We present theoretical analysis and empirical evidence indicating that this approach enhances VLM-based OOD detection performance without any additional training. To further reduce the gap between image and text, we introduce a novel few-shot tuning framework, SUPREME, comprising biased prompts generation (BPG) and image-text consistency (ITC) modules. BPG enhances image-text fusion and improves generalization by conditioning ID text prototypes on the Gaussian-based estimated image domain bias; ITC reduces the modality gap by minimizing intra- and inter-modal distances. Moreover, inspired by our theoretical and empirical findings, we introduce a novel OOD score $S_{\textit{GMP}}$, leveraging uni- and cross-modal similarities. Finally, we present extensive experiments to demonstrate that SUPREME consistently outperforms existing VLM-based OOD detection methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
