arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.14505cs.AIcs.CL
收藏

TaxiNLI:骑上NLU山

TaxiNLI: Taking a Ride up the NLU Hill

Microsoft Research India(微软印度研究院);Google Research(谷歌研究院);University of Richmond(里士满大学)

Pratik Joshi, Somak Aditya, Aalok Sathe, Monojit Choudhury

摘要:

经过预训练的基于transformer的神经架构在自然语言推理(NLI)任务中始终实现了最先进的性能。由于NLI示例包含各种语言、逻辑和推理现象,因此仍不清楚训练系统学习了哪些特定概念,以及它们在哪里可以实现强大的泛化。为了研究这个问题,我们提出了一个分类层次的类别,是相关的NLI任务。我们引入TAXINLI,一个新的数据集,它有来自MNLI数据集的10 k个例子(Williams等人,2018年)与这些分类标签。通过对TAXINLI的各种实验,我们观察到,尽管SOTA神经模型在某些分类类别上已经达到了近乎完美的准确度--比以前的模型有了很大的飞跃--但有些类别仍然很困难。我们的工作增加了越来越多的文献,通过系统地介绍和分析推理类别,显示了当前NLI系统和数据集的差距。

英文摘要:

Pre-trained Transformer-based neural architectures have consistently achieved state-of-the-art performance in the Natural Language Inference (NLI) task. Since NLI examples encompass a variety of linguistic, logical, and reasoning phenomena, it remains unclear as to which specific concepts are learnt by the trained systems and where they can achieve strong generalization. To investigate this question, we propose a taxonomic hierarchy of categories that are relevant for the NLI task. We introduce TAXINLI, a new dataset, that has 10k examples from the MNLI dataset (Williams et al., 2018) with these taxonomic labels. Through various experiments on TAXINLI, we observe that whereas for certain taxonomic categories SOTA neural models have achieved near perfect accuracies - a large jump over the previous models - some categories still remain difficult. Our work adds to the growing body of literature that shows the gaps in the current NLI systems and datasets through a systematic presentation and analysis of reasoning categories.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑