arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.09339cs.CL
收藏

mmJEE-Eval:用于评估视觉-语言模型科学推理能力的双语多模态基准

mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models

Kalinga Institute of Industrial Technology (KIIT)(喀里亚理工学院);Indian Institute of Technology (IIT), Bhubaneswar(印度理工学院(班加罗尔))

Arka Mukherjee, Shreya Ghosh

AI总结:

针对现有基准无法区分VLMs的科学推理与模式匹配能力的问题,本文提出mmJEE-Eval双语多模态基准,通过评估17个模型发现前沿与开源模型的显著差异,能有效区分模型推理能力,公开了相关代码和数据。

摘要:

当代视觉-语言模型(VLMs)在现有多模态推理基准上表现良好(MMMU、MathVista的准确率为78-85%)。然而,这些结果未能充分区分真正的科学推理表达能力与模式匹配能力。为填补这一空白,我们引入了mmJEE-Eval,一个包含1460道题目的双语(英语和印地语)多模态基准,题目来自印度JEE Advanced考试(2019-2025年),涵盖大学预科物理、化学和数学领域。我们对17个最先进模型的评估显示,前沿VLMs(GPT-5、Gemini 2.5 Pro/Flash)在2025年留出题目上达到77-84%的准确率,而开源模型即使扩展到400B参数也停留在37-45%的准确率,这一显著差异在现有基准中未被观察到。尽管谷歌和OpenAI的封闭前沿模型显示出高解题准确率(pass@3分数高达100%),但当推理负载在元认知层面增加时,它们会完全崩溃(GPT-5仅修正5.2%的错误)。系统性消融实验表明,mmJEE-Eval的难度源于复杂性和推理深度而非记忆。实际上,我们的基准在其他基准失效的地方区分了更优的训练和推理方法。我们公开了代码和数据:https://mmjee-eval.github.io

英文摘要:

Contemporary vision-language models (VLMs) perform well on existing multimodal reasoning benchmarks (78-85\% accuracy on MMMU, MathVista). Yet, these results fail to sufficiently distinguish true scientific reasoning articulation capabilities from pattern-matching. To address this gap, we introduce \textbf{mmJEE-Eval}, a multimodal bilingual (English and Hindi) benchmark comprising 1,460 questions from India's JEE Advanced examination (2019-2025) spanning pre-college Physics, Chemistry, and Mathematics domains. Our evaluation of 17 state-of-the-art models reveals that while frontier VLMs (GPT-5, Gemini 2.5 Pro/Flash) achieve 77-84\% accuracy on held-out 2025 questions, open-source models plateau at 37-45\% despite scaling to 400B parameters, a significant difference not observed on existing benchmarks. While closed frontiers from Google and OpenAI show high problem-solving accuracies (up to 100\% pass@3 scores), they fully collapse when the reasoning load is increased meta-cognitively (GPT-5 fixes just 5.2\% errors). Systematic ablations show mmJEE-Eval's difficulty stems from complexity and reasoning depth rather than memorization. Effectively, our benchmark segregates superior training and reasoning methodologies where alternatives fail. We publicly release our code and data: https://mmjee-eval.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑