ProbVLM：面向冻结视觉-语言模型的概率适配器
ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models
AI总结：
本文提出 ProbVLM 概率适配器，为冻结的 CLIP、BLIP 等视觉-语言模型估计多模态嵌入分布与不确定性，并在检索、主动学习和模型选择中验证其优势。
摘要：
诸如 CLIP 之类的大规模视觉-语言模型（VLM）能够成功找到图像与文本之间的对应关系。通过标准的确定性映射过程，图像或文本样本会被映射为嵌入空间中的单一向量。这存在问题：由于多个样本（图像或文本）可能抽象物理世界中的同一概念，确定性嵌入无法反映嵌入空间中固有的歧义。我们提出 ProbVLM，这是一种概率适配器，它以事后方式通过模态间/模态内对齐来估计预训练 VLM 嵌入的概率分布，而无需大规模数据集或计算资源。在 COCO、Flickr、CUB 和 Oxford-flowers 四个具有挑战性的数据集上，我们估计了 CLIP 和 BLIP 两种 VLM 的多模态嵌入不确定性，量化了检索任务中嵌入不确定性的校准，并表明 ProbVLM 优于其他方法。此外，我们提出将主动学习和模型选择作为 VLM 的两项真实下游任务，并表明所估计的不确定性对这两项任务都有帮助。最后，我们提出了一种利用大规模预训练潜在扩散模型可视化嵌入分布的新技术。代码可在 https://github.com/ExplainableML/ProbVLM 获取。
英文摘要：
Large-scale vision-language models (VLMs) like CLIP successfully find correspondences between images and text. Through the standard deterministic mapping process, an image or a text sample is mapped to a single vector in the embedding space. This is problematic: as multiple samples (images or text) can abstract the same concept in the physical world, deterministic embeddings do not reflect the inherent ambiguity in the embedding space. We propose ProbVLM, a probabilistic adapter that estimates probability distributions for the embeddings of pre-trained VLMs via inter/intra-modal alignment in a post-hoc manner without needing large-scale datasets or computing. On four challenging datasets, i.e., COCO, Flickr, CUB, and Oxford-flowers, we estimate the multi-modal embedding uncertainties for two VLMs, i.e., CLIP and BLIP, quantify the calibration of embedding uncertainties in retrieval tasks and show that ProbVLM outperforms other methods. Furthermore, we propose active learning and model selection as two real-world downstream tasks for VLMs and show that the estimated uncertainty aids both tasks. Lastly, we present a novel technique for visualizing the embedding distributions using a large-scale pre-trained latent diffusion model. Code is available at https://github.com/ExplainableML/ProbVLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
