arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.08989cs.CLcs.AI
收藏

大型语言模型对文本扰动的鲁棒性

Robustness of Large Language Models to Perturbations in Text

inQbator AI at eviCore Healthcare Evernorth Health Services(inQbator AI 位于 eviCore 医疗保健 Evernorth 健康服务)

Ayush Singh, Navpreet Singh, Shubham Vatsal

AI总结:

本研究探究大型语言模型对文本噪声扰动的鲁棒性,发现生成式LLMs对形态变化具有韧性,并在语法错误纠正和词汇语义变化基准上达到新最先进水平,同时发布相关数据集和代码。

摘要:

拥有干净的数据集一直是大多数自然语言处理(NLP)系统的基础假设。然而,在现实场景中,规范书写的文本很少见,因此常常使上述基础假设失效。近年来,大型语言模型(LLMs)展示了令人印象深刻的性能,但它们能否处理现实世界数据中不可避免的噪声呢?本研究通过调查LLMs对文本形态变化的韧性来应对这一关键问题。为此,我们人为地在多样化的数据集中引入不同程度的噪声,并系统性地评估LLMs对原始文本损坏变体的鲁棒性。我们的研究结果表明,与普遍看法相反,生成式LLMs对文本中的噪声扰动相当鲁棒。这与预训练模型(如BERT或RoBERTa)形成对比,后者的性能已被证明对恶化的噪声文本敏感。此外,我们在多个密切模拟现实世界中常见错误的真实世界基准上测试了LLMs的韧性。通过最少的提示,LLMs在语法错误纠正(GEC)和词汇语义变化(LSC)基准任务上达到了新的最先进水平。为了赋能未来的研究,我们还发布了一个由人类标注的数据集,表明他们对LLM与人工纠正输出的偏好,以及用于重现我们结果的代码。

英文摘要:

Having a clean dataset has been the foundational assumption of most natural language processing (NLP) systems. However, properly written text is rarely found in real-world scenarios and hence, oftentimes invalidates the aforementioned foundational assumption. Recently, Large language models (LLMs) have shown impressive performance, but can they handle the inevitable noise in real-world data? This work tackles this critical question by investigating LLMs' resilience against morphological variations in text. To that end, we artificially introduce varying levels of noise into a diverse set of datasets and systematically evaluate LLMs' robustness against the corrupt variations of the original text. Our findings show that contrary to popular beliefs, generative LLMs are quiet robust to noisy perturbations in text. This is a departure from pre-trained models like BERT or RoBERTa whose performance has been shown to be sensitive to deteriorating noisy text. Additionally, we test LLMs' resilience on multiple real-world benchmarks that closely mimic commonly found errors in the wild. With minimal prompting, LLMs achieve a new state-of-the-art on the benchmark tasks of Grammar Error Correction (GEC) and Lexical Semantic Change (LSC). To empower future research, we also release a dataset annotated by humans stating their preference for LLM vs. human-corrected outputs along with the code to reproduce our results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑