不只是文本：探索视觉-语言模型中视觉表征的组合性
Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
AI总结：
本文针对视觉-语言模型视觉嵌入空间的组合性不明问题，提出测地可分解嵌入框架，证明预训练VLM视觉表征存在组合结构，在组合分类和群体鲁棒性任务上表现更优，提升了模型可解释性。
摘要：
视觉-语言模型（Vision-Language Models, VLMs）为文本和图像学习共享特征空间，使得不同模态的输入可进行比较。已有研究表明，VLM会将自然语言表征组织为编码组合含义的规则结构，但视觉嵌入空间中是否也会出现组合模式仍不明确。本文研究图像域的组合性，而视觉数据的噪声和稀疏性给组合特性分析带来了挑战。针对这些问题，我们提出了名为测地可分解嵌入（Geodesically Decomposable Embeddings, GDE）的框架，在潜在空间中利用具备几何感知的组合结构近似图像表征。我们证明了预训练VLM的视觉嵌入呈现组合排列，并在组合分类和群体鲁棒性任务中评估了该特性的有效性。与假设潜在空间为线性几何的同类方法相比，GDE在组合分类中取得了更优性能。值得注意的是，它在群体鲁棒性任务中效果尤为突出，结果优于针对特定任务的解决方案。研究结果表明，VLM能够在视觉域自动形成类人的组合推理形式，使其底层过程更具可解释性。代码可在https://github.com/BerasiDavide/vlm_image_compositionality获取。
英文摘要：
Vision-Language Models (VLMs) learn a shared feature space for text and images, enabling the comparison of inputs of different modalities. While prior works demonstrated that VLMs organize natural language representations into regular structures encoding composite meanings, it remains unclear if compositional patterns also emerge in the visual embedding space. In this work, we investigate compositionality in the image domain, where the analysis of compositional properties is challenged by noise and sparsity of visual data. We address these problems and propose a framework, called Geodesically Decomposable Embeddings (GDE), that approximates image representations with geometry-aware compositional structures in the latent space. We demonstrate that visual embeddings of pre-trained VLMs exhibit a compositional arrangement, and evaluate the effectiveness of this property in the tasks of compositional classification and group robustness. GDE achieves stronger performance in compositional classification compared to its counterpart method that assumes linear geometry of the latent space. Notably, it is particularly effective for group robustness, where we achieve higher results than task-specific solutions. Our results indicate that VLMs can automatically develop a human-like form of compositional reasoning in the visual domain, making their underlying processes more interpretable. Code is available at https://github.com/BerasiDavide/vlm_image_compositionality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
