arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.05992cs.CLcs.AI
收藏

PluraMath:将数学推理评估扩展到资源丰富语言之外

PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages

Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Munich Data Science Institute(慕尼黑数据科学研究所);Applied AI Institute(应用人工智能研究所);Charles University(查尔斯大学);Nazarbayev University(纳扎尔拜大学);Inria(法国国家信息与自动化研究所);Indian Institute of Technology, Kharagpur (IIT Kharagpur)(印度Kharagpur理工学院);German University of Digital Science(德国数字科学大学)
Daryna Dementieva, Nikolay Babakov, Kathy Hämmerl, Ilseyar Alimova, Jindřich Libovický, Shu Okabe, Miras Baisbay, Lukas Edman, Abrorkhon Inomkhujaev, Antonia Ka… 展开作者

Daryna Dementieva, Nikolay Babakov, Kathy Hämmerl, Ilseyar Alimova, Jindřich Libovický, Shu Okabe, Miras Baisbay, Lukas Edman, Abrorkhon Inomkhujaev, Antonia Karamolegkou, Mateusz Lango, Volkan Özer, Nikola Selic, Subhankar Swain, Tsedeniya Kinfe Temesgen, Galit Bary Weisberg, Alexander Fraser

AI总结:

研究针对数学推理评估基准偏向高资源语言的问题,引入PluraMath,将其扩展到18种资源不足语言,通过人工策划构建数据集,对27个推理LLMs进行基准测试,分析性能差距,开源相关内容以推动多语言基准开发。

摘要:

数学推理已成为评估和调整推理大语言模型(LLMs)的核心任务,但现有基准严重偏向资源丰富的语言,英语和中文在预训练语料库和评估套件中占主导地位。最近发布的PolyMath数据集是一大进步,但其覆盖范围仍限于18种高资源语言。为填补这一空白,我们引入PluraMath,将PolyMath扩展到另外18种资源不足的语言,涵盖6个语系,从资源中等至极低资源设置。我们通过人工策划管道构建数据集,母语人士彻底验证预计算翻译。然后使用PluraMath对27个推理LLMs在四个模型规模上进行基准测试,探究不同语言条件下的多语言数学推理能力。我们的细粒度分析证实了高资源和资源不足语言在数学推理性能上存在持续差距,更强的结果很大程度上与更好的指令跟随能力相关。我们完全开源数据集、数据采集管道和评估框架,旨在降低资源不足社区进行多语言基准开发的障碍。

英文摘要:

Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑