评估多模态生成模型的组合场景理解能力
Evaluating Compositional Scene Understanding in Multimodal Generative Models
AI总结：
本研究评估了DALL-E 3、GPT-4V等当前多模态生成模型在组合场景理解上的能力，发现其较前代有所提升，但在处理多物体和多重关系的复杂场景时仍远不及人类水平。
摘要：
视觉世界本质上是组合性的。视觉场景由物体及其关系的组合所定义。因此，计算机视觉系统必须反映并利用这种组合性，以实现鲁棒且可泛化的场景理解。尽管在开发通用多模态生成模型（包括文生图模型和多模态视觉-语言模型）方面取得了重大进展，但目前仍不清楚这些系统是否能够准确生成和解释涉及多个物体及其关系组合的场景。在本研究中，我们评估了当前一代文生图模型（DALL-E 3）和多模态视觉-语言模型（GPT-4V、GPT-4o、Claude Sonnet 3.5、QWEN2-VL-72B和InternVL2.5-38B）的组合视觉处理能力，并将这些系统的表现与人类参与者进行了比较。结果表明，这些系统展现出一定的解决组合与关系任务的能力，相较于上一代多模态模型有显著提升，但其性能仍远低于人类参与者的水平，特别是在涉及许多（>5）物体和多重关系的更复杂场景中。这些结果凸显了在视觉场景组合理解方面仍需进一步取得进展。
英文摘要：
The visual world is fundamentally compositional. Visual scenes are defined by the composition of objects and their relations. Hence, it is essential for computer vision systems to reflect and exploit this compositionality to achieve robust and generalizable scene understanding. While major strides have been made toward the development of general-purpose, multimodal generative models, including both text-to-image models and multimodal vision-language models, it remains unclear whether these systems are capable of accurately generating and interpreting scenes involving the composition of multiple objects and relations. In this work, we present an evaluation of the compositional visual processing capabilities in the current generation of text-to-image (DALL-E 3) and multimodal vision-language models (GPT-4V, GPT-4o, Claude Sonnet 3.5, QWEN2-VL-72B, and InternVL2.5-38B), and compare the performance of these systems to human participants. The results suggest that these systems display some ability to solve compositional and relational tasks, showing notable improvements over the previous generation of multimodal models, but with performance nevertheless well below the level of human participants, particularly for more complex scenes involving many ($>5$) objects and multiple relations. These results highlight the need for further progress toward compositional understanding of visual scenes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
