MMReason：面向AGI的MLLM开放式多模态多步推理基准
MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
AI总结：
针对现有基准在评估长链推理方面的不足，本文提出MMReason基准，通过多领域多难度的开放式问题、多模型投票过滤及三元评分机制，全面精确评估多模态大语言模型的多步推理能力。
摘要：
推理在推动多模态大语言模型（MLLMs）迈向通用人工智能（AGI）方面发挥着至关重要的作用。然而，现有的MLLM基准在精确且全面地评估长链推理能力方面，往往在以下三个关键方面存在不足：（1）缺乏难度和多样性；（2）易受猜测和记忆的影响；（3）对中间推理步骤的评估不够充分。为了填补这一空白，我们引入了MMReason，这是一个旨在通过多样化、开放式、具有挑战性的问题来精确且全面评估MLLM长链推理能力的新基准。首先，我们从多个领域（即6个学科）和多个难度级别（即从大学前到大学，从基础到竞赛级别）中筛选出需要多步推理的挑战性问题。其次，我们将这些问题重新表述为开放式格式，并使用多模型投票技术进行过滤，以消除与猜测和记忆相关的捷径情况，从而确保稳健的推理评估。第三，我们为问题标注了详细的逐步解答，并设计了一种基于参考的三元评分机制，以可靠地评估中间推理步骤。借助MMReason，我们对流行的领先MLLMs进行了基准测试，并对其推理能力进行了深入分析。我们希望MMReason能成为推进MLLM推理研究的宝贵资源。代码将在 https://github.com/HJYao00/MMReason 上提供。
英文摘要：
Reasoning plays a crucial role in advancing Multimodal Large Language Models (MLLMs) toward Artificial General Intelligence. However, existing MLLM benchmarks often fall short in precisely and comprehensively evaluating long-chain reasoning abilities from three key aspects: (1) lack of difficulty and diversity, (2) susceptibility to guessability and memorization, (3) inadequate assessment of intermediate reasoning steps. To fill this gap, we introduce MMReason, a new benchmark designed to precisely and comprehensively evaluate MLLM long-chain reasoning capability with diverse, open-ended, challenging questions. First, we curate challenging questions requiring multi-step reasoning from various fields (i.e., 6 disciplines) and multiple difficulty levels (i.e., from pre-university to university, and from foundational to competition tiers). Second, these questions are reformulated into an open-ended format and filtered using a multi-model voting technique to eliminate shortcut cases related to guessing and memorization, ensuring robust reasoning evaluations. Third, we annotate the questions with detailed step-by-step solutions, and design a reference-based ternary scoring mechanism to reliably assess intermediate reasoning steps. With MMReason, we benchmark popular leading MLLMs and provide an in-depth analysis of their reasoning capabilities. We hope MMReason will serve as a valuable resource for advancing MLLM reasoning research. Code will be available at https://github.com/HJYao00/MMReason.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
