零样本VLM中虚假相关性的密度感知转换
Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs
AI总结：
提出密度感知转换（DAT）方法，利用局部几何密度项修正图像-文本相似度，以缓解CLIP等视觉语言模型在零样本分类中因虚假相关性导致的性能下降。
摘要：
视觉语言模型（如CLIP）实现了强大的零样本分类。然而，它们的预测仍然对虚假相关性敏感，即上下文线索主导语义内容。早期的解决方案通常依赖于微调或提示工程，这要么削弱了预训练模型的优势，要么容易产生幻觉。在这项工作中，我们提出了密度感知转换（DAT），它使用从组参考集导出的局部几何密度项来细化图像-文本相似度分数。我们的方法受到以下现象的启发：CLIP嵌入表现出模态间隙，并位于特征空间中的各向异性壳上：常见模式聚集在均值附近，而罕见模式被推向外围。这种几何结构产生了不均匀的对齐，其中虚假相关性被放大，而语义上有意义但罕见的线索被边缘化。为了解决这个问题，我们采用相对度量根据嵌入密度重新缩放相似度，抑制扩散区域中过度自信的分数，同时保留密集、语义一致的匹配。在基准数据集上的实验结果表明，最差组和平均准确率持续提高，突出了密度感知转换作为一种简单有效的校准机制，用于使用多模态模型进行可靠的零样本分类。
英文摘要：
Vision-Language models (VLMs), such as CLIP, achieve powerful zero-shot classification. However, their predictions remain sensitive to spurious correlations, where contextual cues dominate over semantic content. Earlier solutions typically rely on fine-tuning or prompt engineering, which either undermine the advantages of pre-trained models or are prone to hallucination. In this work, we propose Density-Aware Translation (DAT) that refines image-text similarity scores using a local geometric density term derived from group reference sets. Our approach is motivated by the phenomenon that CLIP embeddings exhibit a modality gap and lie on an anisotropic shell in the feature space: common patterns cluster near the mean, while rare patterns are pushed outward. This geometry creates uneven alignment, where spurious correlations are amplified while semantically meaningful but rare cues are marginalised. To address this, we employ a relative measure to rescale similarities based on embedding density, suppressing overconfident scores in diffuse regions while preserving dense, semantically consistent matches. Experimental results on benchmark datasets demonstrate consistent improvements in worst-group and average accuracy, highlighting density-aware translation as a simple and effective calibration mechanism for reliable zero-shot classification using multimodal models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
