BMMR:一个大规模双语多模态多学科推理数据集
BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
AI总结:
本文提出BMMR,一个包含11万道大学水平题目的双语多模态多学科推理数据集,并配套BMMR-Verifier验证器,实验表明现有模型仍有较大提升空间,微调可缩小差距。
摘要:
在本文中,我们介绍了BMMR,一个大规模的双语、多模态、多学科推理数据集,供社区开发和评估大型多模态模型(LMMs)。BMMR包含11万个大学水平的问题,涵盖300个由联合国教科文组织定义的学科,格式多样——包括选择题、填空题和开放式问答——数据来源于印刷和数字媒体,如书籍、考试和测验。所有数据通过人机协同和可扩展的框架进行整理和过滤,每个实例都配有高质量的推理路径。该数据集分为两部分:BMMR-Eval包含20,458个高质量实例,用于全面评估LMMs在中文和英文下跨多个学科的知识和推理能力;BMMR-Train包含88,991个实例,以支持进一步的研究和开发,将当前对数学推理的关注扩展到多样化的学科和领域。此外,我们提出了基于过程的多学科验证器(即BMMR-Verifier),用于对推理路径进行准确和细粒度的评估。对24个模型的广泛实验表明:(i)即使是最先进的模型(如o3和Gemini-2.5-Pro)在BMMR-Eval上仍有很大的提升空间;(ii)推理模型表现出学科偏差,仅在特定学科上优于LMMs;(iii)开源模型仍然落后于其专有对应模型;(iv)在BMMR-Train上进行微调缩小了这一差距。此外,我们使用BMMR-Verifier和其他深入研究进行了推理链分析,揭示了LMMs目前在多学科推理中面临的挑战。我们将发布数据,并希望我们的工作能为社区提供见解和贡献。
英文摘要:
In this paper, we introduce BMMR, a large-scale bilingual, multimodal, multi-disciplinary reasoning dataset for the community to develop and evaluate large multimodal models (LMMs). BMMR comprises 110k college-level questions spanning 300 UNESCO-defined subjects, spanning diverse formats-multiple-choice, fill-in-the-blank, and open-ended QA-and sourced from both print and digital media such as books, exams, and quizzes. All data are curated and filtered via a human-in-the-loop and scalable framework, and each instance is paired with a high-quality reasoning path. The dataset is organized into two parts: BMMR-Eval that comprises 20,458 high-quality instances to comprehensively assess LMMs' knowledge and reasoning across multiple disciplines in both Chinese and English; and BMMR-Train that contains 88,991 instances to support further research and development, extending the current focus on mathematical reasoning to diverse disciplines and domains. In addition, we propose the process-based multi-discipline verifier (i.e., BMMR-Verifier) for accurate and fine-grained evaluation of reasoning paths. Extensive experiments on 24 models reveal that (i) even SOTA models (e.g., o3 and Gemini-2.5-Pro) leave substantial headroom on BMMR-Eval; (ii) reasoning models exhibit discipline bias and outperform LMMs only on specific subjects; (iii) open-source models still trail their proprietary counterparts; and (iv) fine-tuning on BMMR-Train narrows this gap. Additionally, we conduct reasoning-chain analyses using BMMR-Verifier and other in-depth studies, uncovering the challenges LMMs currently face in multidisciplinary reasoning. We will release the data, and we hope our work can offer insights and contributions to the community.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。