arXiv 2504.057822025-04-08cs.CVcs.AI

MDK12-Bench：一个用于评估多模态大语言模型推理能力的多学科基准

MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models

Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；USTC（中国科学技术大学）；RIT（罗切斯特理工学院）；HIT（哈尔滨工业大学）；WHU（武汉大学）；MBZUAI（Mohamed bin Zayed 人工智能大学）；NUS（新加坡国立大学）

Pengfei Zhou, Fanrui Zhang, Xiaopeng Peng, Zhaopan Xu, Jiaxin Ai, Yansheng Qiu, Chuanhao Li, Zhen Li, Ming Li, Yukang Feng, Jianwen Sun, Haoquan Zhang, Zizhen L… 展开作者

Pengfei Zhou, Fanrui Zhang, Xiaopeng Peng, Zhaopan Xu, Jiaxin Ai, Yansheng Qiu, Chuanhao Li, Zhen Li, Ming Li, Yukang Feng, Jianwen Sun, Haoquan Zhang, Zizhen Li, Xiaofeng Mao, Wangbo Zhao, Kai Wang, Xiaojun Chang, Wenqi Shao, Yang You, Kaipeng Zhang

备注：: 11 pages, 8 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MDK12-Bench多学科基准，利用14万道K-12真实试题及知识点标注和动态评估框架，系统评估并揭示现有多模态大语言模型在推理能力上的显著不足。

摘要：

多模态推理将语言和视觉线索整合到问题解决与决策中，是人类智能的基本方面，也是迈向通用人工智能的关键一步。然而，对多模态大语言模型（MLLMs）多模态推理能力的评估仍然不足。现有的大多数推理基准受限于数据规模有限、领域覆盖范围狭窄以及知识分布缺乏结构化。为填补这些空白，我们提出了MDK12-Bench，一个通过真实世界K-12考试评估MLLMs推理能力的多学科基准。该基准涵盖六个学科（数学、物理、化学、生物、地理和信息科学），包含14万个推理实例，覆盖从小学到12年级的不同难度级别。它基于组织良好的知识结构提供了6827个实例级知识点标注、详细的答案解释、难度标签和跨年级划分，为全面评估提供了稳健的平台。此外，我们提出了一种新颖的动态评估框架，通过在评估过程中对问题形式、问题类型和图像风格进行自举式变换来缓解数据污染问题。在MDK12-Bench上的大量实验揭示了当前MLLMs在多模态推理方面的显著局限。我们在该基准上的发现为下一代模型的开发提供了见解。我们的数据和代码可在https://github.com/LanceZPF/MDK12获取。

英文摘要：

Multimodal reasoning, which integrates language and visual cues into problem solving and decision making, is a fundamental aspect of human intelligence and a crucial step toward artificial general intelligence. However, the evaluation of multimodal reasoning capabilities in Multimodal Large Language Models (MLLMs) remains inadequate. Most existing reasoning benchmarks are constrained by limited data size, narrow domain coverage, and unstructured knowledge distribution. To close these gaps, we introduce MDK12-Bench, a multi-disciplinary benchmark assessing the reasoning capabilities of MLLMs via real-world K-12 examinations. Spanning six disciplines (math, physics, chemistry, biology, geography, and information science), our benchmark comprises 140K reasoning instances across diverse difficulty levels from primary school to 12th grade. It features 6,827 instance-level knowledge point annotations based on a well-organized knowledge structure, detailed answer explanations, difficulty labels and cross-year partitions, providing a robust platform for comprehensive evaluation. Additionally, we present a novel dynamic evaluation framework to mitigate data contamination issues by bootstrapping question forms, question types, and image styles during evaluation. Extensive experiment on MDK12-Bench reveals the significant limitation of current MLLMs in multimodal reasoning. The findings on our benchmark provide insights into the development of the next-generation models. Our data and codes are available at https://github.com/LanceZPF/MDK12.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。