arXiv 2507.118822025-07-16cs.CL

Marco-Bench-MIF：关于大型语言模型的多语言指令遵循能力

Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language Models

Alibaba International Digital Commerce（阿里巴巴国际数字商务）；University of Aberdeen（阿伯丁大学）；MBZUAI

Bo Zeng, Chenyang Lyu, Sinuo Liu, Mingyan Zeng, Minghao Wu, Xuanfan Ni, Tianqi Shi, Yu Zhao, Yefeng Liu, Chenyu Zhu, Ruizhe Li, Jiahui Geng, Qing Li, Yu Tong, L… 展开作者

Bo Zeng, Chenyang Lyu, Sinuo Liu, Mingyan Zeng, Minghao Wu, Xuanfan Ni, Tianqi Shi, Yu Zhao, Yefeng Liu, Chenyu Zhu, Ruizhe Li, Jiahui Geng, Qing Li, Yu Tong, Longyue Wang, Weihua Luo, Kaifu Zhang

备注：: ACL 2025 Main Conference paper

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出了多语言指令遵循基准Marco-Bench-MIF，通过结合翻译与验证的混合流水线对IFEval进行本地化扩展，涵盖30种语言，全面评估揭示了高低资源语言间的准确率差距、模型规模影响及机器翻译数据的低估问题。

摘要：

指令遵循能力已成为大型语言模型（LLM）需要评估的主要能力。然而，现有的数据集（如IFEval）要么主要以英语为中心的单语种，要么只是机器翻译成其他语言，限制了它们在多语言环境中的适用性。本文提出了IFEval的一个精心策划的扩展，将其本地化为名为Marco-Bench-MIF的多语言版本，涵盖30种具有不同本地化程度的语言。我们的基准通过结合翻译与验证的混合流水线，解决了语言约束（例如修改中文的大写要求）和文化引用（例如在提示中替换特定地区的公司名称）问题。通过在Marco-Bench-MIF上对20多个LLM进行全面评估，我们发现：（1）高/低资源语言之间存在25-35%的准确率差距；（2）模型规模在很大程度上影响了45-60%的性能，但特定书写系统的挑战依然存在；（3）与本地化数据相比，机器翻译数据低估了7-22%的准确率。我们的分析指出了多语言指令遵循面临的挑战，包括跨语言的关键词一致性保持和组合约束遵守。Marco-Bench-MIF可在https://github.com/AIDC-AI/Marco-Bench-MIF获取。

英文摘要：

Instruction-following capability has become a major ability to be evaluated for Large Language Models (LLMs). However, existing datasets, such as IFEval, are either predominantly monolingual and centered on English or simply machine translated to other languages, limiting their applicability in multilingual contexts. In this paper, we present an carefully-curated extension of IFEval to a localized multilingual version named Marco-Bench-MIF, covering 30 languages with varying levels of localization. Our benchmark addresses linguistic constraints (e.g., modifying capitalization requirements for Chinese) and cultural references (e.g., substituting region-specific company names in prompts) via a hybrid pipeline combining translation with verification. Through comprehensive evaluation of 20+ LLMs on our Marco-Bench-MIF, we found that: (1) 25-35% accuracy gap between high/low-resource languages, (2) model scales largely impact performance by 45-60% yet persists script-specific challenges, and (3) machine-translated data underestimates accuracy by7-22% versus localized data. Our analysis identifies challenges in multilingual instruction following, including keyword consistency preservation and compositional constraint adherence across languages. Our Marco-Bench-MIF is available at https://github.com/AIDC-AI/Marco-Bench-MIF.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。