arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.17476cs.CLcs.AI

MultiNRC：面向 LLM 的具有挑战性的原生多语言推理评估基准

MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs

Scale AI

Alexander R. Fabbri, Diego Mares, Jorge Flores, Meher Mankikar, Ernesto Hernandez, Dean Lee, Bing Liu, Chen Xing

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 MultiNRC 原生多语言推理基准，含千余道法语、西班牙语和中文文化根植问题，并评估 14 个 LLM，揭示其原生推理与跨文化数学能力仍不足。

摘要：

尽管近期的大型语言模型（LLM）在英语推理基准上表现出快速提升，但对此类 LLM 跨越不同语言和文化语境的多语言推理能力的评估仍然有限。现有的多语言推理基准通常通过翻译现有英语推理基准构建，使这些基准偏向以英语语言/文化为语境的推理问题。在本工作中，我们推出 Multilingual Native Reasoning Challenge（MultiNRC，多语言原生推理挑战），这一基准旨在通过法语、西班牙语和中文母语者撰写的 1,000 多个原生、语言及文化根植的推理问题来评估 LLM。MultiNRC 涵盖四个核心推理类别：特定语言的语言推理、文字游戏与谜语、文化/传统推理，以及具有文化相关性的数学推理。对于文化/传统推理和具有文化相关性的数学推理，我们还由精通英语的母语者通过人工翻译提供多语言问题的英语对等翻译。这套英语对等内容可以直接比较 LLM 在其他语言与英语上处理相同推理问题时的推理能力。我们在 MultiNRC 及其英语对等集上系统评估了当前 14 个领先 LLM，覆盖大多数 LLM 系列。结果表明：（1）当前 LLM 仍不擅长原生多语言推理，没有模型在 MultiNRC 上得分超过 50%；（2）LLM 在处理语言、文化和逻辑推理任务时表现出明显不同的优势与弱点；（3）大多数模型在英语数学推理上的表现显著优于原始语言（+10%），表明文化根植知识方面仍存在持续挑战。

英文摘要：

Although recent Large Language Models (LLMs) have shown rapid improvement on reasoning benchmarks in English, the evaluation of such LLMs' multilingual reasoning capability across diverse languages and cultural contexts remains limited. Existing multilingual reasoning benchmarks are typically constructed by translating existing English reasoning benchmarks, biasing these benchmarks towards reasoning problems with context in English language/cultures. In this work, we introduce the Multilingual Native Reasoning Challenge (MultiNRC), a benchmark designed to assess LLMs on more than 1,000 native, linguistic and culturally grounded reasoning questions written by native speakers in French, Spanish, and Chinese. MultiNRC covers four core reasoning categories: language-specific linguistic reasoning, wordplay & riddles, cultural/tradition reasoning, and math reasoning with cultural relevance. For cultural/tradition reasoning and math reasoning with cultural relevance, we also provide English equivalent translations of the multilingual questions by manual translation from native speakers fluent in English. This set of English equivalents can provide a direct comparison of LLM reasoning capacity in other languages vs. English on the same reasoning questions. We systematically evaluate current 14 leading LLMs covering most LLM families on MultiNRC and its English equivalent set. The results show that (1) current LLMs are still not good at native multilingual reasoning, with none scoring above 50% on MultiNRC; (2) LLMs exhibit distinct strengths and weaknesses in handling linguistic, cultural, and logical reasoning tasks; (3) Most models perform substantially better in math reasoning in English compared to in original languages (+10%), indicating persistent challenges with culturally grounded knowledge.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑