PluraMath:将数学推理评估扩展到资源丰富语言之外
PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
AI总结:
研究针对数学推理评估基准偏向高资源语言的问题,引入PluraMath,将其扩展到18种资源不足语言,通过人工策划构建数据集,对27个推理LLMs进行基准测试,分析性能差距,开源相关内容以推动多语言基准开发。
摘要:
数学推理已成为评估和调整推理大语言模型(LLMs)的核心任务,但现有基准严重偏向资源丰富的语言,英语和中文在预训练语料库和评估套件中占主导地位。最近发布的PolyMath数据集是一大进步,但其覆盖范围仍限于18种高资源语言。为填补这一空白,我们引入PluraMath,将PolyMath扩展到另外18种资源不足的语言,涵盖6个语系,从资源中等至极低资源设置。我们通过人工策划管道构建数据集,母语人士彻底验证预计算翻译。然后使用PluraMath对27个推理LLMs在四个模型规模上进行基准测试,探究不同语言条件下的多语言数学推理能力。我们的细粒度分析证实了高资源和资源不足语言在数学推理性能上存在持续差距,更强的结果很大程度上与更好的指令跟随能力相关。我们完全开源数据集、数据采集管道和评估框架,旨在降低资源不足社区进行多语言基准开发的障碍。
英文摘要:
Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。