NLI数据健全性检查：评估数据损坏对模型性能的影响
NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance
摘要：
预训练的神经语言模型在自然语言推理（NLI）任务上具有高性能。但他们是否真的理解处理过的序列的含义仍不清楚。我们提出了一个新的诊断测试套件，它允许评估数据集是否构成一个很好的测试平台，用于评估模型的意义理解能力。我们专门将受控腐败转换应用于广泛使用的基准测试（MNLI和ANLI），这涉及删除整个单词类，并经常导致无意义的句子对。如果受损数据的模型准确性仍然很高，那么数据集可能包含指导预测的统计偏差和伪影。然而，模型精度的大幅下降表明原始数据集对模型的推理能力提出了适当的挑战。因此，我们提出的控制可以作为一个碰撞测试，为NLI任务开发高质量的数据。
英文摘要：
Pre-trained neural language models give high performance on natural language inference (NLI) tasks. But whether they actually understand the meaning of the processed sequences remains unclear. We propose a new diagnostics test suite which allows to assess whether a dataset constitutes a good testbed for evaluating the models' meaning understanding capabilities. We specifically apply controlled corruption transformations to widely used benchmarks (MNLI and ANLI), which involve removing entire word classes and often lead to non-sensical sentence pairs. If model accuracy on the corrupted data remains high, then the dataset is likely to contain statistical biases and artefacts that guide prediction. Inversely, a large decrease in model accuracy indicates that the original dataset provides a proper challenge to the models' reasoning capabilities. Hence, our proposed controls can serve as a crash test for developing high quality data for NLI tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
