MOAT：评估LMMs在能力整合和指令接地方面的表现
MOAT: Evaluating LMMs for Capability Integration and Instruction Grounding
AI总结：
MOAT通过1005个复杂现实视觉问题评估LMMs在能力整合和指令接地方面的表现，发现最佳模型仅达44%准确率，揭示了LMMs在复杂任务中的不足。
摘要：
大型多模态模型（LMMs）在视觉-语言（VL）任务中展示了显著的潜力。然而，LMMs在现实任务中的应用受到阻碍，因为它们在需要结合VL能力的任务以及涉及复杂文本或视觉指令接地的任务中表现不佳。为了彻底研究这一差距及其根本原因，我们提出了MOAT，一个包含1005个复杂的现实视觉问题的多样化基准，这些问题对人类来说很简单，但对LMMs来说却具有挑战性。具体而言，MOAT中的任务要求LMMs通过整合VL能力（如阅读文本、计数、理解空间关系、接地文本和视觉指令等）进行通用问题解决。所有这些能力都符合我们提出的分类学，该分类学包含9种VL能力，使MOAT能够提供对LMMs的优缺点的细粒度视图。此外，MOAT是第一个明确评估LMMs接地复杂文本和视觉指令能力的基准，这对于许多现实应用至关重要。我们评估了17个专有和开源的LMMs，发现表现最好的LMM（Gemini 2.5 Pro）仅达到44%的准确率，远低于现实应用中可接受的水平。为了指导未来模型开发，我们分析了我们结果中的常见趋势，并讨论了性能不佳的根本原因，重点在于文本中心推理的影响，这在复杂任务中成为VL能力的瓶颈，以及平铺的潜在有害影响。代码和数据可在https://cambrian-yzt.github.io/MOAT/上获取。
英文摘要：
Large multimodal models (LMMs) have demonstrated significant potential as generalists in vision-language (VL) tasks. However, adoption of LMMs in real-world tasks is hindered by their poor performance in tasks that require a combination of VL capabilities, as well as in tasks that involve the grounding of complex text or visual instructions. To thoroughly investigate this gap and its underlying causes, we propose MOAT, a diverse benchmark with 1005 complex real-world vision questions that are straightforward for humans but challenging for LMMs. Specifically, the tasks in MOAT require LMMs to engage in generalist problem solving by integrating VL capabilities such as reading text, counting, understanding spatial relations, grounding textual and visual instructions, etc. All these abilities fit into a taxonomy proposed by us that contains 9 VL capabilities, enabling MOAT to provide a fine-grained view of LMMs' strengths and weaknesses. Besides, MOAT is the first benchmark to explicitly evaluate LMMs' ability to ground complex text and visual instructions, which is essential for many real-world applications. We evaluated 17 proprietary and open source LMMs, finding that the best performing LMM (Gemini 2.5 Pro) achieved only 44% accuracy, far below what would be acceptable in real-world applications. To guide future model development, we analyze common trends in our results and discuss the underlying causes of poor performance, focusing on the impact of text-centric reasoning, which VL capabilities form bottlenecks in complex tasks, and the potential harmful effects of tiling. Code and data are available at https://cambrian-yzt.github.io/MOAT/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
