用多模态多图像推理基准评估多模态大语言模型
Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
AI总结：
本文提出首个多图像结构化视觉推理基准MMRB，包含92个子任务及多解法思维链标注，并通过句子级匹配框架评估40个MLLMs，揭示开源模型与商业模型在多图像推理上的显著差距。
摘要：
随着能力的增强和应用的广泛普及，多模态大语言模型（MLLMs）越来越多地被要求同时处理并推理多张图像。然而，现有的MLLM基准要么专注于单图像视觉推理，要么只对多图像理解任务进行最终答案评估，使得MLLMs在多图像输入上的推理能力在很大程度上未被充分探索。为弥补这一空白，我们引入了多模态多图像推理基准（MMRB），这是首个旨在评估跨多张图像的结构化视觉推理的基准。MMRB包含92个子任务，涵盖空间、时间和语义推理，并带有由GPT-4o生成、经人类专家精炼的多解法、思维链（CoT）风格标注。同时设计了一个衍生子集，用于在多图像场景中评估多模态奖励模型。为支持快速且可扩展的评估，我们提出了一个使用开源大语言模型的句子级匹配框架。在40个MLLMs（包括9个推理专用模型和8个奖励模型）上的广泛基线实验表明，开源MLLMs在多图像推理任务中仍显著落后于商业MLLMs。此外，当前的多模态奖励模型几乎无法处理多图像奖励排序任务。
英文摘要：
With enhanced capabilities and widespread applications, Multimodal Large Language Models (MLLMs) are increasingly required to process and reason over multiple images simultaneously. However, existing MLLM benchmarks focus either on single-image visual reasoning or on multi-image understanding tasks with only final-answer evaluation, leaving the reasoning capabilities of MLLMs over multi-image inputs largely underexplored. To address this gap, we introduce the $\textbf{Multimodal Multi-image Reasoning Benchmark (MMRB)}$, the first benchmark designed to evaluate structured visual reasoning across multiple images. MMRB comprises $\textbf{92 sub-tasks}$ covering spatial, temporal, and semantic reasoning, with multi-solution, CoT-style annotations generated by GPT-4o and refined by human experts. A derivative subset is designed to evaluate multimodal reward models in multi-image scenarios. To support fast and scalable evaluation, we propose a sentence-level matching framework using open-source LLMs. Extensive baseline experiments on $\textbf{40 MLLMs}$, including 9 reasoning-specific models and 8 reward models, demonstrate that open-source MLLMs still lag significantly behind commercial MLLMs in multi-image reasoning tasks. Furthermore, current multimodal reward models are nearly incapable of handling multi-image reward ranking tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
