MGSM-Pro：一种用于多语言数学推理评估的简单策略
MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
AI总结：
本文提出MGSM-Pro，通过增加数字变化实例提升多语言数学推理评估的鲁棒性，发现低资源语言在不同数字实例下表现下降，且专有模型对数字变化更敏感。
摘要：
大型语言模型在数学推理方面取得了显著进展。然而，多语言评估的基准开发在难度和时效性上都滞后于英语。最近，GSM-Symbolic展示了当模型在不同问题实例上进行评估时高方差的证据；然而，该评估仅在英语中进行。在本文中，我们引入MGSM-Pro，这是MGSM数据集的扩展，采用了GSM-Symbolic方法。我们的数据集通过改变名称、数字和无关上下文，为每个MGSM问题提供五个实例。跨九种语言的评估表明，许多低资源语言在测试集中的数字实例与原始测试集不同的情况下会出现较大的性能下降。我们进一步发现，在HRL设置中模型的鲁棒性并不必然适用于LRL。此外，专有模型，如Gemini 2.5 Flash和GPT-4.1对数字的鲁棒性较低，而Gemini 3.0 Pro则更鲁棒。在开放模型中，GPT-OSS 120B和DeepSeek v3显示出更强的鲁棒性。基于这些发现，我们建议对每个问题至少使用五个数字变化的实例进行评估，以获得更鲁棒和现实的数学推理评估。
英文摘要：
Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency. Recently, GSM-Symbolic showed a strong evidence of high variance when models are evaluated on different instantiations of the same question; however, the evaluation was conducted only in English. In this paper, we introduce MGSM-Pro, an extension of MGSM dataset with GSM-Symbolic approach. Our dataset provides five instantiations per MGSM question by varying names, digits and irrelevant context. Evaluations across nine languages reveal that many low-resource languages suffer large performance drops when tested on digit instantiations different from those in the original test set. We further find that models robustness in HRL setting do not necessarily translate to LRL. Moreover, proprietary models, such as Gemini 2.5 Flash and GPT-4.1 are less robust to digit, whereas Gemini 3.0 Pro is more robust. Among open models, GPT-OSS 120B and DeepSeek v3 show stronger robustness. Based on these findings, we recommend evaluating each problem using at least five digit-varying instantiations to obtain a more robust and realistic assessment of math reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
