如果CLIP能说话：通过其偏好的概念描述理解视觉-语言模型表示
If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions
AI总结：
针对VLMs表示是否优先视觉属性的问题，提出EX2方法通过强化学习对齐LLM与VLM偏好生成重要文本特征描述，发现虚假描述和非视觉属性显著影响VLM表示，且不同VLMs属性优先级不同。
摘要：
近期研究常假设视觉-语言模型（VLM）的表示基于形状等视觉属性，但VLMs在多大程度上优先利用这类信息表示概念尚不明确。我们提出一种名为Extract and Explore（EX2）的新方法，用于表征对VLMs重要的文本特征。EX2利用强化学习将大语言模型与VLM偏好对齐，并生成包含对VLM重要特征的描述。随后，我们通过检查这些描述来识别对VLM表示有贡献的特征。使用EX2，我们发现尽管虚假描述（如“Click to enlarge photo of CONCEPT”）未提供有用信息，但它们在VLM表示中仍占主要作用。更重要的是，在信息性描述中，VLMs显著依赖栖息地（如北美）等非视觉属性来表示视觉概念。此外，我们的分析显示不同VLMs在其表示中优先考虑不同的属性。总体而言，我们表明VLMs并非简单地将图像与场景描述匹配，非视觉甚至虚假描述会显著影响其表示。
英文摘要：
Recent works often assume that Vision-Language Model (VLM) representations are based on visual attributes like shape. However, it is unclear to what extent VLMs prioritize this information to represent concepts. We propose Extract and Explore (EX2), a novel approach to characterize textual features that are important for VLMs. EX2 uses reinforcement learning to align a large language model with VLM preferences and generates descriptions that incorporate features that are important for the VLM. Then, we inspect the descriptions to identify features that contribute to VLM representations. Using EX2, we find that spurious descriptions have a major role in VLM representations despite providing no helpful information, e.g., Click to enlarge photo of CONCEPT. More importantly, among informative descriptions, VLMs rely significantly on non-visual attributes like habitat (e.g., North America) to represent visual concepts. Also, our analysis reveals that different VLMs prioritize different attributes in their representations. Overall, we show that VLMs do not simply match images to scene descriptions and that non-visual or even spurious descriptions significantly influence their representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
