跨模态掩码组合概念建模以增强视觉-语言组合性
Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality
AI总结：
提出MACCO框架，通过掩码一个模态的组合概念并从另一模态完整上下文重建，增强视觉-语言模型的组合理解能力，在五个基准上显著提升。
摘要：
对比训练的视觉-语言模型（如CLIP）在学习联合图像-文本表示方面取得了显著进展，但在组合理解方面仍面临挑战。它们通常表现出“词袋”行为——难以捕捉对象关系、属性-对象绑定和词序依赖。这一限制不仅源于优化时依赖全局单向量表示，还源于对配对图像文本数据中固有丰富组合信息的利用和建模不足。在这项工作中，我们提出了MACCO（掩码组合概念建模）框架，该框架掩码一个模态中的组合概念，并基于另一模态的完整上下文信息重建它们，从而使模型能够更有效地捕捉和对齐跨模态组合结构。为促进这一过程，我们引入了两个辅助目标，在模态间和模态内联合对齐和正则化掩码特征。在五个组合基准上的大量实验和深入分析表明，我们的方法不仅显著增强了VLM的组合性，还提高了它们捕捉句法结构和语言信息的能力。此外，改进的组合性也有利于文本到图像生成和多模态大语言模型。代码可在https://this URL获取。
英文摘要：
Contrastively trained vision-language models like CLIP, have made remarkable progress in learning joint image-text representations, but still face challenges in compositional understanding. They often exhibit a "bag-of-words" behavior--struggling to capture the object relations, attribute-object bindings, and word order dependencies. This limitation arises not only from the reliance on global, single-vector representations for optimization, but also from the insufficient exploitation and modeling of the rich compositional information inherently present in paired image text data. In this work, we propose MACCO (MAsked Compositional Concept MOdeling), a framework that masks compositional concepts in one modality and reconstructs them conditioned on the full contextual information from the other, enabling the model to capture and align cross-modal compositional structures more effectively. To facilitate this process, we introduce two auxiliary objectives that jointly align and regularize masked features both inter-modally and intra-modally. Extensive experiments on five compositional benchmarks, along with in-depth analyses, demonstrate that our approach not only significantly enhances compositionality in VLMs but also improves their ability to capture syntactic structure and linguistic information. Additionally, the improved compositionality also benefits text-to-image generation and multimodal large language model. Code is available at https://github.com/hiker-lw/MACCO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
