概念区域至关重要：采用新型聚类重要性方法对CLIP进行基准测试
Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach
AI总结：
针对CLIP等视觉语言模型易受虚假关联影响的问题，提出CCI可解释性方法与COVAR基准，结合二者全面评估18种CLIP变体，为构建更鲁棒的VLM提供支撑。
摘要：
CLIP等对比视觉语言模型（VLM）虽具备出色的零样本识别能力，却易受虚假关联影响，尤其存在过度依赖背景的问题。本文提出基于聚类的概念重要性（CCI）这一新型可解释性方法，该方法利用CLIP自身的patch嵌入将空间patch分组为语义连贯的聚类，对其进行掩码后评估模型预测的相对变化。CCI在忠实度基准上达到了新的当前最优水平，大幅超越此前方法；例如在MS COCO检索的deletion-AUC指标上，其性能提升超过两倍。本文进一步提出，CCI与GroundedSAM结合时，可自动将预测分类为前景驱动或背景驱动，提供关键的诊断能力。然而CounterAnimals等现有基准仅依赖准确率，默认将所有性能下降都归因于背景关联。本文分析表明该假设并不完备，因为许多误差来自视角变化、尺度偏移和细粒度物体混淆。为厘清这些影响，本文提出COVAR基准，可系统性地改变物体前景与背景。借助CCI与COVAR，本文对18种CLIP变体开展了全面评估，所提供的方法进展与实证证据为构建更鲁棒的VLM指明了方向。
英文摘要：
Contrastive vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition yet remain vulnerable to spurious correlations, particularly background over-reliance. We introduce Cluster-based Concept Importance (CCI), a novel interpretability method that uses CLIP's own patch embeddings to group spatial patches into semantically coherent clusters, mask them, and evaluate relative changes in model predictions. CCI sets a new state of the art on faithfulness benchmarks, surpassing prior methods by large margins; for example, it yields more than a twofold improvement on the deletion-AUC metric for MS COCO retrieval. We further propose that CCI, when combined with GroundedSAM, automatically categorizes predictions as foreground- or background-driven, providing a crucial diagnostic ability. Existing benchmarks such as CounterAnimals, however, rely solely on accuracy and implicitly attribute all performance degradation to background correlations. Our analysis shows this assumption to be incomplete, since many errors arise from viewpoint variation, scale shifts, and fine-grained object confusions. To disentangle these effects, we introduce COVAR, a benchmark that systematically varies object foregrounds and backgrounds. Leveraging CCI with COVAR, we present a comprehensive evaluation of eighteen CLIP variants, offering methodological advances and empirical evidence that chart a path toward more robust VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
