arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.03483cs.CLcs.AI
收藏

BMMR:一个大规模双语多模态多学科推理数据集

BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset

Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);East China Normal University(华东师范大学);Oxford(牛津大学);University of Sydney(悉尼大学);Yimudata(云墨数据)
Zhiheng Xi, Guanyu Li, Yutao Fan, Honglin Guo, Yufang Liu, Xiaoran Fan, Jiaqi Liu, Jingchao Ding, Wangmeng Zuo, Zhenfei Yin, Lei Bai, Tao Ji, Tao Gui, Qi Zhang,… 展开作者

Zhiheng Xi, Guanyu Li, Yutao Fan, Honglin Guo, Yufang Liu, Xiaoran Fan, Jiaqi Liu, Jingchao Ding, Wangmeng Zuo, Zhenfei Yin, Lei Bai, Tao Ji, Tao Gui, Qi Zhang, Philip Torr, Xuanjing Huang

AI总结:

本文提出BMMR,一个包含11万道大学水平题目的双语多模态多学科推理数据集,并配套BMMR-Verifier验证器,实验表明现有模型仍有较大提升空间,微调可缩小差距。

摘要:

在本文中,我们介绍了BMMR,一个大规模的双语、多模态、多学科推理数据集,供社区开发和评估大型多模态模型(LMMs)。BMMR包含11万个大学水平的问题,涵盖300个由联合国教科文组织定义的学科,格式多样——包括选择题、填空题和开放式问答——数据来源于印刷和数字媒体,如书籍、考试和测验。所有数据通过人机协同和可扩展的框架进行整理和过滤,每个实例都配有高质量的推理路径。该数据集分为两部分:BMMR-Eval包含20,458个高质量实例,用于全面评估LMMs在中文和英文下跨多个学科的知识和推理能力;BMMR-Train包含88,991个实例,以支持进一步的研究和开发,将当前对数学推理的关注扩展到多样化的学科和领域。此外,我们提出了基于过程的多学科验证器(即BMMR-Verifier),用于对推理路径进行准确和细粒度的评估。对24个模型的广泛实验表明:(i)即使是最先进的模型(如o3和Gemini-2.5-Pro)在BMMR-Eval上仍有很大的提升空间;(ii)推理模型表现出学科偏差,仅在特定学科上优于LMMs;(iii)开源模型仍然落后于其专有对应模型;(iv)在BMMR-Train上进行微调缩小了这一差距。此外,我们使用BMMR-Verifier和其他深入研究进行了推理链分析,揭示了LMMs目前在多学科推理中面临的挑战。我们将发布数据,并希望我们的工作能为社区提供见解和贡献。

英文摘要:

In this paper, we introduce BMMR, a large-scale bilingual, multimodal, multi-disciplinary reasoning dataset for the community to develop and evaluate large multimodal models (LMMs). BMMR comprises 110k college-level questions spanning 300 UNESCO-defined subjects, spanning diverse formats-multiple-choice, fill-in-the-blank, and open-ended QA-and sourced from both print and digital media such as books, exams, and quizzes. All data are curated and filtered via a human-in-the-loop and scalable framework, and each instance is paired with a high-quality reasoning path. The dataset is organized into two parts: BMMR-Eval that comprises 20,458 high-quality instances to comprehensively assess LMMs' knowledge and reasoning across multiple disciplines in both Chinese and English; and BMMR-Train that contains 88,991 instances to support further research and development, extending the current focus on mathematical reasoning to diverse disciplines and domains. In addition, we propose the process-based multi-discipline verifier (i.e., BMMR-Verifier) for accurate and fine-grained evaluation of reasoning paths. Extensive experiments on 24 models reveal that (i) even SOTA models (e.g., o3 and Gemini-2.5-Pro) leave substantial headroom on BMMR-Eval; (ii) reasoning models exhibit discipline bias and outperform LMMs only on specific subjects; (iii) open-source models still trail their proprietary counterparts; and (iv) fine-tuning on BMMR-Train narrows this gap. Additionally, we conduct reasoning-chain analyses using BMMR-Verifier and other in-depth studies, uncovering the challenges LMMs currently face in multidisciplinary reasoning. We will release the data, and we hope our work can offer insights and contributions to the community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑