适配对比语言-图像预训练（CLIP）模型用于分布外检测
Adapting Contrastive Language-Image Pretrained (CLIP) Models for Out-of-Distribution Detection
AI总结：
本研究聚焦CLIP模型的OOD检测适配，提出伪标签探测方法，在5个ImageNet基准上超越SOTA，同时发现线性探测优于微调，且大参数CLIP仍难检测对抗OOD样本。
摘要：
我们针对用于视觉分布外（OOD）检测的预训练特征提取器开展了全面的实验研究，重点是适配对比语言-图像预训练（CLIP）模型。在未对训练数据进行微调的情况下，我们在4个基准测试中证实了CLIP模型的分布内分类与无监督OOD检测之间存在正相关关系（$R^2\neq0.92$）。我们进一步提出了一种名为\textit{伪标签探测}（pseudo-label probing, PLP）的新型简单可扩展方法，用于适配视觉-语言模型以实现OOD检测。给定训练集的一组标签名称，PLP利用从CLIP文本编码器得到的伪标签训练一个线性层。为测试预训练模型的OOD检测鲁棒性，我们提出了一种基于特征的新型对抗式OOD数据操纵方法来生成对抗样本。有趣的是，我们发现：（i）在所有5个基于ImageNet的大规模基准测试中，PLP的表现均优于此前的最优方法\textit{ming2022mcm}，使用最大的CLIP模型（ViT-G）时，AUROC平均提升了3.4%；（ii）对于CLIP架构，线性探测的表现大幅优于微调（例如CLIP ViT-H在所有基于ImageNet的基准测试中平均AUROC提升7.3%）；（iii）十亿参数级的CLIP模型仍无法检测出经对抗操纵的OOD图像。代码和对抗生成的数据集将公开发布。
英文摘要：
We present a comprehensive experimental study on pretrained feature extractors for visual out-of-distribution (OOD) detection, focusing on adapting contrastive language-image pretrained (CLIP) models. Without fine-tuning on the training data, we are able to establish a positive correlation ($R^2\geq0.92$) between in-distribution classification and unsupervised OOD detection for CLIP models in $4$ benchmarks. We further propose a new simple and scalable method called \textit{pseudo-label probing} (PLP) that adapts vision-language models for OOD detection. Given a set of label names of the training set, PLP trains a linear layer using the pseudo-labels derived from the text encoder of CLIP. To test the OOD detection robustness of pretrained models, we develop a novel feature-based adversarial OOD data manipulation approach to create adversarial samples. Intriguingly, we show that (i) PLP outperforms the previous state-of-the-art \citep{ming2022mcm} on all $5$ large-scale benchmarks based on ImageNet, specifically by an average AUROC gain of 3.4\% using the largest CLIP model (ViT-G), (ii) we show that linear probing outperforms fine-tuning by large margins for CLIP architectures (i.e. CLIP ViT-H achieves a mean gain of 7.3\% AUROC on average on all ImageNet-based benchmarks), and (iii) billion-parameter CLIP models still fail at detecting adversarially manipulated OOD images. The code and adversarially created datasets will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
