人类对齐基准：MLLMs与人类推理能力的细粒度评估
Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans
AI总结：
提出Human-Aligned Bench基准，含9,794个多模态推理问题，细粒度评估MLLMs与人类推理能力，发现显著差异，为下一代模型开发提供见解。
摘要：
实现通用人工智能（AGI）的目标是模仿人类并超越人类。OpenAI的o1、o3以及DeepSeek的R1等模型已经证明，具有类人推理能力的大语言模型（LLMs）表现出卓越的性能，并正逐步融入多模态大语言模型（MLLMs）中。然而，这些模型在处理推理任务时是否具备与人类相当的能力，目前尚不清楚。在本文中，我们提出了Human-Aligned Bench，一个用于多模态推理与人类表现细粒度对齐的基准。具体而言，我们收集了9,794个仅依赖上下文推理的多模态问题，包括双语（中文和英文）多模态问题和纯文本问题，涵盖四种问题类型：视觉推理、定义判断、类比推理和逻辑判断。更重要的是，每个问题都附带人类成功率以及人类容易错误选择的选项。在Human-Aligned Bench上进行的大量实验揭示了当前MLLMs在多模态推理中的表现与人类表现之间存在显著差异。我们的基准上的发现为下一代模型的开发提供了见解。
英文摘要：
The goal of achieving Artificial General Intelligence (AGI) is to imitate humans and surpass them. Models such as OpenAI's o1, o3, and DeepSeek's R1 have demonstrated that large language models (LLMs) with human-like reasoning capabilities exhibit exceptional performance and are being gradually integrated into multimodal large language models (MLLMs). However, whether these models possess capabilities comparable to humans in handling reasoning tasks remains unclear at present. In this paper, we propose Human-Aligned Bench, a benchmark for fine-grained alignment of multimodal reasoning with human performance. Specifically, we collected 9,794 multimodal questions that solely rely on contextual reasoning, including bilingual (Chinese and English) multimodal questions and pure text-based questions, encompassing four question types: visual reasoning, definition judgment, analogical reasoning, and logical judgment. More importantly, each question is accompanied by human success rates and options that humans are prone to choosing incorrectly. Extensive experiments on the Human-Aligned Bench reveal notable differences between the performance of current MLLMs in multimodal reasoning and human performance. The findings on our benchmark provide insights into the development of the next-generation models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
