arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.22019cs.CV
收藏

类内概率嵌入用于视觉-语言模型中的不确定性估计

Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models

Queensland University of Technology(昆士兰理工大学)

Zhenxiang Lin, Maryam Haghighat, Will Browne, Dimity Miller

AI总结:

本研究提出一种无需训练的后处理方法,通过类内概率嵌入提升视觉-语言模型的不确定性估计,有效检测错误预测。

摘要:

视觉-语言模型(VLMs),如CLIP,因其强大的开放词汇分类性能而受到欢迎,但它们容易将错误分类赋予高置信度分数,限制了其在安全关键应用中的可靠性。我们介绍了一种无需训练的后处理不确定性估计方法,用于对比学习VLMs,可用于检测错误预测。我们的方法关键在于测量类内视觉特征的一致性,通过特征投影结合多元高斯分布来创建类特定的概率嵌入。我们的方法是VLM无关的,不需要微调,对分布偏移具有鲁棒性,并且即使每类仅使用10张训练图像也能有效工作。在ImageNet、Flowers102、Food101、EuroSAT和DTD上的广泛实验表明,我们的方法在错误检测性能上达到了最先进的水平,显著优于确定性和概率性VLM基线。代码可在https://github.com/zhenxianglin/ICPE上获取。

英文摘要:

Vision-language models (VLMs), such as CLIP, have gained popularity for their strong open vocabulary classification performance, but they are prone to assigning high confidence scores to misclassifications, limiting their reliability in safety-critical applications. We introduce a training-free, post-hoc uncertainty estimation method for contrastive VLMs that can be used to detect erroneous predictions. The key to our approach is to measure visual feature consistency within a class, using feature projection combined with multivariate Gaussians to create class-specific probabilistic embeddings. Our method is VLM-agnostic, requires no fine-tuning, demonstrates robustness to distribution shift, and works effectively with as few as 10 training images per class. Extensive experiments on ImageNet, Flowers102, Food101, EuroSAT and DTD show state-of-the-art error detection performance, significantly outperforming both deterministic and probabilistic VLM baselines. Code is available at https://github.com/zhenxianglin/ICPE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑