arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.09536cs.CL
收藏

评估大型语言模型对多语言拼写错误的鲁棒性

Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors

Center for Information and Language Processing(信息与语言处理中心);LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心(MCML))

Raoyuan Zhao, Yihong Liu, Lena Altinger, Hinrich Schütze, Michael A. Hedderich

AI总结:

本文提出MulTypo算法,评估18种开源LLM在多语言拼写错误下的表现,发现拼写错误显著降低性能,尤其在生成任务和推理任务中,且语言和任务类型影响鲁棒性,强调需关注噪声训练和多语言鲁棒性评估。

摘要:

大型语言模型(LLMs)越来越多地部署在多语言、真实世界应用中,用户输入中自然引入拼写错误(typos)。然而,大多数基准测试假设输入干净,导致LLMs在跨语言拼写错误下的鲁棒性研究不足。为此,我们引入MulTypo,一种多语言拼写生成算法,基于语言特定的键盘布局和打字行为模拟人类样式的错误。我们评估了18种开源LLM,涵盖三种模型家族和五个下游任务,包括语言推理、多选问答、数学推理和机器翻译。结果表明,拼写错误一致降低性能,特别是在生成任务和需要推理的任务中,而自然语言推理任务相对更鲁棒。指令微调提升干净输入性能,但可能在噪声下增加脆弱性。我们还观察到语言依赖性鲁棒性:高资源语言通常比低资源语言更鲁棒,且从英语翻译比翻译成英语更鲁棒。我们的发现强调了噪声感知训练和多语言鲁棒性评估的必要性。我们发布了MulTypo的Python包,并在https://github.com/cisnlp/multypo公开源代码。

英文摘要:

Large language models (LLMs) are increasingly deployed in multilingual, real-world applications with user inputs -- naturally introducing \emph{typographical errors} (typos). Yet most benchmarks assume clean input, leaving the robustness of LLMs to typos across languages largely underexplored. To address this gap, we introduce MulTypo, a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. We evaluate 18 open-source LLMs across three model families and five downstream tasks spanning language inference, multi-choice question answering, mathematical reasoning, and machine translation tasks. Our results show that typos consistently degrade performance, particularly in generative tasks and those requiring reasoning -- while the natural language inference task is comparatively more robust. Instruction tuning improves clean-input performance but may increase brittleness under noise. We also observe language-dependent robustness: high-resource languages are generally more robust than low-resource ones, and translation from English is more robust than translation into English. Our findings underscore the need for noise-aware training and multilingual robustness evaluation. We release a Python package for MulTypo and make the source code publicly available at https://github.com/cisnlp/multypo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑