arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.04723cs.CLcs.AI
收藏

英语作为第二语言与拼写错误的个体与综合影响

Individual and Combined Effects of English as a Second Language and Typos on LLM Performance

Harvard University(哈佛大学)

Serena Liu, Yutong Yang, Prisha Sheth, Weixuan Dong, Mingjiao Diao, Xinru Zhu, Nikhil Banga, Oscar Melendez, Arnav Sharma, Minda Zhao, Marina Lin, Mengyu Wang

AI总结:

研究探讨了英语作为第二语言和拼写错误对大语言模型性能的影响,发现两者结合时性能下降更显著,且在封闭式任务中表现更一致,而开放式任务结果更混杂。

摘要:

大型语言模型(LLMs)在全球范围内被使用,由于其训练数据主要为英语,因此在英语输入上表现最佳。因此,许多非英语母语者以英语为第二语言(ESL)与它们交互,这些输入常包含拼写错误。先前研究主要单独研究了ESL变体和拼写错误的影响,尽管它们在现实使用中常同时出现。在本研究中,我们使用Trans-EnV框架将标准英语输入转换为八个ESL变体,并应用MulTypo在三个级别(低、中、高)注入拼写错误。我们发现,结合ESL变体和拼写错误通常导致比单独因素更大的性能下降,尽管这种综合影响并非简单相加。这种模式在封闭式任务中最为明显,其中性能下降可以更一致地跨ESL变体和拼写错误级别进行表征,而开放式任务的结果则更为混杂。总体而言,这些发现表明,在干净的标准英语上评估可能高估了现实中的模型性能,并且单独评估ESL变体和拼写错误无法完全捕捉模型在现实环境中的行为。

英文摘要:

Large language models (LLMs) are used globally, and because much of their training data is in English, they typically perform best on English inputs. As a result, many non-native English speakers interact with them in English as a second language (ESL), and these inputs often contain typographical errors. Prior work has largely studied the effects of ESL variation and typographical errors separately, even though they often co-occur in real-world use. In this study, we use the Trans-EnV framework to transform standard English inputs into eight ESL variants and apply MulTypo to inject typos at three levels: low, moderate, and severe. We find that combining ESL variation and typos generally leads to larger performance drops than either factor alone, though the combined effect is not simply additive. This pattern is clearest on closed-ended tasks, where performance degradation can be characterized more consistently across ESL variants and typo levels, while results on open-ended tasks are more mixed. Overall, these findings suggest that evaluations on clean standard English may overestimate real-world model performance, and that evaluating ESL variation and typographical errors in isolation does not fully capture model behavior in realistic settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑