arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1810.09774cs.CL
收藏

在NLI基准测试中测试神经网络模型的泛化能力

Testing the Generalization Power of Neural Network Models Across NLI Benchmarks

University of Helsinki(赫尔辛基大学);Basement AI(Basement人工智能);University of Gothenburg(哥德堡大学)

Aarne Talman, Stergios Chatzikyriakidis

摘要:

神经网络模型在自然语言推理中非常成功,在一些基准测试中,最好的模型达到90%的准确率。然而,这些模型的成功在很大程度上取决于具体的基准。我们发现,在从一个基准测试中提取的自然语言推理数据集上训练的模型在其他基准测试中表现不佳,即使这些基准测试中假设的推理概念相同或相似。我们训练六个高性能的神经网络模型在不同的数据集,并表明,这些都有问题的推广时,我们用一个测试集从另一个语料库设计的相同的任务取代原来的测试集。根据这些结果,我们认为,目前的大多数神经网络模型是不能够推广以及在自然语言推理任务。我们发现,当数据集足够相似时,使用大型预训练语言模型有助于迁移学习。我们的研究结果还强调,目前的NLI数据集没有足够广泛地覆盖推理的不同细微差别。

英文摘要:

Neural network models have been very successful in natural language inference, with the best models reaching 90% accuracy in some benchmarks. However, the success of these models turns out to be largely benchmark specific. We show that models trained on a natural language inference dataset drawn from one benchmark fail to perform well in others, even if the notion of inference assumed in these benchmarks is the same or similar. We train six high performing neural network models on different datasets and show that each one of these has problems of generalizing when we replace the original test set with a test set taken from another corpus designed for the same task. In light of these results, we argue that most of the current neural network models are not able to generalize well in the task of natural language inference. We find that using large pre-trained language models helps with transfer learning when the datasets are similar enough. Our results also highlight that the current NLI datasets do not cover the different nuances of inference extensively enough.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑