扩展项目反应理论以实现高效且有意义的多语言评估
Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation
AI总结：
提出Multilingual-IRT框架，通过引入每语言难度偏差、分离内容与语言效应的区分度及每语言能力残差，解决多语言基准测试中的线性扩展、翻译错误和文化特定知识混淆问题，在MMLU-Pro-X上实现更优的预测和错误检测。
摘要：
多语言基准测试对于评估跨语言的大语言模型（LLMs）至关重要，但它们存在三个问题：详尽评估随语言数量线性增长，自动翻译引入的错误在大规模下容易被忽略，以及某些项目混淆了通用知识和文化特定知识。我们通过一个统一的统计框架Multilingual-IRT来解决这三个问题，该框架扩展了项目反应理论，引入了每语言难度偏差、分离内容与语言效应的区分度以及每语言能力残差。在MMLU-Pro-X的29种语言上对25个LLM拟合Multilingual-IRT，我们表明其拟合参数支持三种实际应用：预测未观察到的（项目、LLM、语言）实例，其二元交叉熵比最强的基于准确率的基线低11-16%；发现分布在所有28种非英语语言中的候选翻译错误，而基于准确率的基线将检测集中在少数语言上；以及恢复基于准确率的基线遗漏的文化特定项目。
英文摘要：
Multilingual benchmarks are central to evaluating large language models (LLMs) across languages, but they suffer from three issues: exhaustive evaluation scales linearly with the number of languages, automatic translation introduces errors that are easily missed at scale, and some items conflate general and culture-specific knowledge. We address all three with a unified statistical framework, Multilingual-IRT, which extends Item Response Theory with per-language difficulty deviations, split discriminability separating content from language effects, and per-language ability residuals. Fitting Multilingual-IRT on 25 LLMs across 29 languages of MMLU-Pro-X, we show that its fitted parameters support three practical applications: predicting unobserved (item, LLM, language) instances with 11-16% lower binary cross-entropy than the strongest accuracy-based baseline, surfacing candidate translation errors distributed across all 28 non-English languages, whereas accuracy-based baselines concentrate detections in a few languages, and recovering culture-specific items that accuracy-based baselines miss.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
