arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.12942cs.CLcs.AIcs.CY
收藏

生成数据以减轻自然语言推理数据集中的伪相关性

Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets

University College London(伦敦大学学院);Microsoft Semantic Machines(微软语义机器公司);Allen Institute for AI(艾伦人工智能研究所)

Yuxiang Wu, Matt Gardner, Pontus Stenetorp, Pradeep Dasigi

摘要:

自然语言处理模型通常利用数据集中与任务无关的特征和标签之间的虚假相关性,仅在它们所训练的分布中表现良好,而不能推广到不同的任务分布。我们建议通过生成数据集的去偏版本来解决这个问题,然后可以通过简单地替换其训练数据来训练去偏的现成模型。我们的方法包括:1)一种用于训练数据生成器生成高质量、标签一致的数据样本的方法; 2)一种过滤机制,用于去除导致虚假相关性的数据点,以z统计量衡量。我们生成SNLI和MNLI数据集的去偏版本,并在一个大型的去偏、分布外和对抗性测试集上进行评估。结果表明,在我们的去偏数据集上训练的模型在所有设置中都比在原始数据集上训练的模型更好。在大多数数据集上,我们的方法优于或执行先前最先进的去偏置策略,并且当与正交技术、专家产品相结合时,它进一步改进并优于SNLI硬和MNLI硬的先前最佳结果。

英文摘要:

Natural language processing models often exploit spurious correlations between task-independent features and labels in datasets to perform well only within the distributions they are trained on, while not generalising to different task distributions. We propose to tackle this problem by generating a debiased version of a dataset, which can then be used to train a debiased, off-the-shelf model, by simply replacing its training data. Our approach consists of 1) a method for training data generators to generate high-quality, label-consistent data samples; and 2) a filtering mechanism for removing data points that contribute to spurious correlations, measured in terms of z-statistics. We generate debiased versions of the SNLI and MNLI datasets, and we evaluate on a large suite of debiased, out-of-distribution, and adversarial test sets. Results show that models trained on our debiased datasets generalise better than those trained on the original datasets in all settings. On the majority of the datasets, our method outperforms or performs comparably to previous state-of-the-art debiasing strategies, and when combined with an orthogonal technique, product-of-experts, it improves further and outperforms previous best results of SNLI-hard and MNLI-hard.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑