MMCOMPOSITION：重新审视预训练视觉-语言模型的组合性
MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
AI总结：
本文提出人工标注基准 MMCOMPOSITION，全面评估视觉-语言模型在对象交互、计数等深层推理上的组合性，实验揭示 GPT-4o 等模型在细粒度组合感知与推理上的局限性。
摘要：
大型视觉-语言模型（VLMs）的出现显著推进了多模态理解，使得在各种任务中能够更复杂、更准确地整合视觉和文本信息，这些任务包括图像和视频描述、视觉问答以及跨模态检索。尽管 VLMs 具有卓越的能力，但研究人员对其组合性——即理解和生成已知视觉与文本组件新组合的能力——缺乏全面的理解。先前的基准测试仅从对象、关系和属性的角度提供了相对粗略的组合性评估，而忽略了对对象交互、计数和复杂组合的深层推理。然而，组合性是 VLMs 实现跨模态连贯推理与理解的关键能力。为解决这一局限，我们提出了 MMCOMPOSITION，一个全新的人工标注基准，用于全面且准确地评估 VLMs 的组合性。我们提出的基准作为对这些早期工作的补充。借助 MMCOMPOSITION，我们能够量化和探索主流 VLMs 的组合性。令人惊讶的是，我们发现 GPT-4o 的组合性劣于最佳的开源模型，并分析了其深层原因。我们的实验分析揭示了 VLMs 在细粒度组合感知与推理方面的局限性，并指明了 VLM 设计与训练的改进方向。资源获取地址：https://hanghuacs.github.io/MMComposition/
英文摘要：
The advent of large Vision-Language Models (VLMs) has significantly advanced multimodal understanding, enabling more sophisticated and accurate integration of visual and textual information across various tasks, including image and video captioning, visual question answering, and cross-modal retrieval. Despite VLMs' superior capabilities, researchers lack a comprehensive understanding of their compositionality -- the ability to understand and produce novel combinations of known visual and textual components. Prior benchmarks provide only a relatively rough compositionality evaluation from the perspectives of objects, relations, and attributes while neglecting deeper reasoning about object interactions, counting, and complex compositions. However, compositionality is a critical ability that facilitates coherent reasoning and understanding across modalities for VLMs. To address this limitation, we propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively and accurately evaluating VLMs' compositionality. Our proposed benchmark serves as a complement to these earlier works. With MMCOMPOSITION, we can quantify and explore the compositionality of the mainstream VLMs. Surprisingly, we find GPT-4o's compositionality inferior to the best open-source model, and we analyze the underlying reasons. Our experimental analysis reveals the limitations of VLMs in fine-grained compositional perception and reasoning, and points to areas for improvement in VLM design and training. Resources available at: https://hanghuacs.github.io/MMComposition/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
