反事实增强的SNLI训练数据并不比未增强的数据产生更好的泛化
Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data
摘要:
越来越多的工作表明,模型利用注释工件来实现标准众包基准测试的最新性能-从众包工作者收集的数据集来创建评估任务-同时仍然失败于同一任务的域外示例。最近的工作已经探索了使用反事实增强数据--通过最小化编辑一组种子示例以产生反事实标签而构建的数据--来增强与这些基准相关的训练数据,并构建更鲁棒的分类器。然而,Khashabi等人(2020)发现,当控制数据集大小和收集成本时,这种类型的增强对阅读理解任务几乎没有好处。我们在这项工作的基础上,使用英语自然语言推理数据来测试模型的泛化和鲁棒性,并发现在反事实增强的SNLI数据集上训练的模型并不比类似大小的未增强数据集更好地泛化,并且反事实增强可能会损害性能,产生对挑战性示例不太鲁棒的模型。通过标准众包技术对自然语言理解数据进行反事实增强似乎不是收集训练数据的有效方式,需要进一步创新才能使这一工作可行。
英文摘要:
A growing body of work shows that models exploit annotation artifacts to achieve state-of-the-art performance on standard crowdsourced benchmarks---datasets collected from crowdworkers to create an evaluation task---while still failing on out-of-domain examples for the same task. Recent work has explored the use of counterfactually-augmented data---data built by minimally editing a set of seed examples to yield counterfactual labels---to augment training data associated with these benchmarks and build more robust classifiers that generalize better. However, Khashabi et al. (2020) find that this type of augmentation yields little benefit on reading comprehension tasks when controlling for dataset size and cost of collection. We build upon this work by using English natural language inference data to test model generalization and robustness and find that models trained on a counterfactually-augmented SNLI dataset do not generalize better than unaugmented datasets of similar size and that counterfactual augmentation can hurt performance, yielding models that are less robust to challenge examples. Counterfactual augmentation of natural language understanding data through standard crowdsourcing techniques does not appear to be an effective way of collecting training data and further innovation is required to make this general line of work viable.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。