arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.07324cs.CL
收藏

READIN:一个具有真实且多样输入噪声的中文多任务基准

READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises

Tsinghua University(清华大学);University of Maryland, College Park(马里兰大学帕克分校);International Innovation Center of Tsinghua University(清华大学国际创新中心)

Chenglei Si, Zhengyan Zhang, Yingfa Chen, Xiaozhi Wang, Zhiyuan Liu, Maosong Sun

AI总结:

本文构建了中文多任务基准 READIN,通过拼音和语音输入引入真实多样的输入噪声,实验表明现有预训练模型即使使用鲁棒训练方法仍会显著性能下降。

摘要:

在许多现实应用中,用户生成的输入通常包含各种噪声,这些噪声源于语言差异引起的语音识别错误或打字错误。因此,在具有真实输入噪声的数据上测试模型性能以确保鲁棒性和公平性至关重要。然而,针对中文构建此类基准的研究很少,而中文在现实世界中存在各种语言特有的输入噪声。为了填补这一重要空白,我们构建了 READIN:一个具有真实且多样输入噪声的中文多任务基准。READIN 包含四个不同的任务,并要求标注者使用两种常用的中文输入法(拼音输入和语音输入)重新输入原始测试数据。我们设计了标注流程以最大化多样性,例如指示标注者使用不同的输入法编辑器来产生键盘噪声,并招募来自不同方言群体的说话人来产生语音噪声。我们在一系列强大的预训练语言模型以及鲁棒训练方法上进行了实验,发现即使在采用数据增强等鲁棒性方法的情况下,这些模型在 READIN 上的性能也经常出现显著下降。作为创建面向用户生成输入的噪声基准的首次大规模尝试,我们相信 READIN 是对现有中文 NLP 基准的重要补充。源代码和数据集可从 https://github.com/thunlp/READIN 获取。

英文摘要:

For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations1 or typographical errors (typos). Thus, it is crucial to test model performance on data with realistic input noises to ensure robustness and fairness. However, little study has been done to construct such benchmarks for Chinese, where various language-specific input noises happen in the real world. In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input. We designed our annotation pipeline to maximize diversity, for example by instructing the annotators to use diverse input method editors (IMEs) for keyboard noises and recruiting speakers from diverse dialectical groups for speech noises. We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN even with robustness methods like data augmentation. As the first large-scale attempt in creating a benchmark with noises geared towards user-generated inputs, we believe that READIN serves as an important complement to existing Chinese NLP benchmarks. The source code and dataset can be obtained from https://github.com/thunlp/READIN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑