多样化您的数据集:通过对抗数据集的受控方差分析泛化
Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets
摘要:
现象特定的“对抗性”数据集最近被设计为针对特定推理类型执行有针对性的压力测试。最近的工作(Liu et al.,2019 a)提出,这样的数据集可以用于训练NLI和其他类型的模型,通常允许学习焦点现象并改进挑战数据集,这表明原始训练数据中存在“盲点”。然而,尽管模型可以在这样的训练过程中得到改进,但它仍然可能容易受到针对相同现象但来自不同分布的其他挑战数据集的影响,例如具有不同的语法复杂性水平。在这项工作中,我们扩展了这种方法,通过控制对抗数据集中的其他方面,来得出关于模型学习和概括目标现象的能力的结论,而不是“学习”数据集。我们展示了我们的方法上的两个推理现象-与格交替和数值推理,阐述,并在某些情况下矛盾,刘等人的结果。我们的方法能够构建更好的挑战数据集,以创建更强大的模型,并可能产生更好的模型理解和随后的总体改进。
英文摘要:
Phenomenon-specific "adversarial" datasets have been recently designed to perform targeted stress-tests for particular inference types. Recent work (Liu et al., 2019a) proposed that such datasets can be utilized for training NLI and other types of models, often allowing to learn the phenomenon in focus and improve on the challenge dataset, indicating a "blind spot" in the original training data. Yet, although a model can improve in such a training process, it might still be vulnerable to other challenge datasets targeting the same phenomenon but drawn from a different distribution, such as having a different syntactic complexity level. In this work, we extend this method to drive conclusions about a model's ability to learn and generalize a target phenomenon rather than to "learn" a dataset, by controlling additional aspects in the adversarial datasets. We demonstrate our approach on two inference phenomena - dative alternation and numerical reasoning, elaborating, and in some cases contradicting, the results of Liu et al.. Our methodology enables building better challenge datasets for creating more robust models, and may yield better model understanding and subsequent overarching improvements.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。