BERT能容忍多少噪声？来自多语言句子难度检测的见解
How Much Noise Can BERT Handle? Insights from Multilingual Sentence Difficulty Detection
AI总结：
研究探讨了BERT在噪声下的鲁棒性，通过多语言句子难度检测验证了去噪策略的效果，发现GMM在小数据集上效果显著，而大语料库中预训练模型的正则化更具优势。
摘要：
嘈杂的训练数据会显著降低基于语言模型的分类器性能，特别是在非主题分类任务中。在本研究中，我们设计了一个方法框架来评估去噪的影响。更具体地说，我们探索了多种去噪策略用于句子难度检测，使用的训练数据来源于通过嘈杂众包获取的文档级难度注释。除了单语言设置外，我们还处理了跨语言迁移问题，即在一种语言中训练多语言语言模型并在另一种语言中进行测试。我们评估了几种去噪技术，包括高斯混合模型（GMM）、共教、噪声转移矩阵和标签平滑。我们的结果表明，尽管基于BERT的模型对噪声具有固有的鲁棒性，但加入显式的噪声检测可以进一步提高性能。对于我们的较小数据集，基于GMM的噪声过滤在提高预测质量方面特别有效，通过将曲线下面积分数从0.52提高到0.92，或在结合去噪方法时提高到0.93。然而，对于我们的较大数据集，预训练语言模型的内在正则化提供了强大的基线，去噪方法仅带来微小的提升（从0.92到0.94，而两种去噪方法的组合没有贡献）。不过，移除嘈杂句子（约占数据集的20%）有助于生成一个更干净的语料库，其中包含更少的不妥之处。因此，我们发布了最大的多语言语料库用于句子难度预测：见https://github.com/Nouran-Khallaf/denoising-difficulty
英文摘要：
Noisy training data can significantly degrade the performance of language-model-based classifiers, particularly in non-topical classification tasks. In this study we designed a methodological framework to assess the impact of denoising. More specifically, we explored a range of denoising strategies for sentence-level difficulty detection, using training data derived from document-level difficulty annotations obtained through noisy crowdsourcing. Beyond monolingual settings, we also address cross-lingual transfer, where a multilingual language model is trained in one language and tested in another. We evaluate several noise reduction techniques, including Gaussian Mixture Models (GMM), Co-Teaching, Noise Transition Matrices, and Label Smoothing. Our results indicate that while BERT-based models exhibit inherent robustness to noise, incorporating explicit noise detection can further enhance performance. For our smaller dataset, GMM-based noise filtering proves particularly effective in improving prediction quality by raising the Area-Under-the-Curve score from 0.52 to 0.92, or to 0.93 when de-noising methods are combined. However, for our larger dataset, the intrinsic regularisation of pre-trained language models provides a strong baseline, with denoising methods yielding only marginal gains (from 0.92 to 0.94, while a combination of two denoising methods made no contribution). Nonetheless, removing noisy sentences (about 20\% of the dataset) helps in producing a cleaner corpus with fewer infelicities. As a result we have released the largest multilingual corpus for sentence difficulty prediction: see https://github.com/Nouran-Khallaf/denoising-difficulty
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
