arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2002.06261cs.AI
收藏

自然语言理解任务中基于transformer模型的压力测试评估

Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks

IMFD

Carlos Aspillaga, Andrés Carvallo, Vladimir Araujo

摘要:

由于Transformer架构的引入,近年来自然语言处理领域取得了重大进展。目前最先进的模型,通过大量的参数和大量的文本语料库上的预训练,已经显示出令人印象深刻的结果在几个下游任务。许多研究人员研究了以前的(非Transformer)模型,以了解它们在不同场景下的实际行为,表明这些模型正在利用数据集的线索或故障,并且输入数据的轻微扰动会严重降低它们的性能。相比之下,最近的模型还没有进行系统的测试,以显示其在严重的压力条件下的鲁棒性与敌对的例子。出于这个原因,这项工作在自然语言推理(NLI)和问答(QA)任务中评估了三个基于transformer的模型(RoBERTa,XLNet和BERT),以了解它们是否更健壮,或者它们是否与它们的前辈有相同的缺陷。实验结果表明,对于NLI和QA任务,RoBERTa、XLNet和BERT模型比递归神经网络模型具有更强的抗压力测试能力。尽管如此,它们仍然非常脆弱,并表现出各种意想不到的行为,从而揭示了该领域仍有未来改进的空间。

英文摘要:

There has been significant progress in recent years in the field of Natural Language Processing thanks to the introduction of the Transformer architecture. Current state-of-the-art models, via a large number of parameters and pre-training on massive text corpus, have shown impressive results on several downstream tasks. Many researchers have studied previous (non-Transformer) models to understand their actual behavior under different scenarios, showing that these models are taking advantage of clues or failures of datasets and that slight perturbations on the input data can severely reduce their performance. In contrast, recent models have not been systematically tested with adversarial-examples in order to show their robustness under severe stress conditions. For that reason, this work evaluates three Transformer-based models (RoBERTa, XLNet, and BERT) in Natural Language Inference (NLI) and Question Answering (QA) tasks to know if they are more robust or if they have the same flaws as their predecessors. As a result, our experiments reveal that RoBERTa, XLNet and BERT are more robust than recurrent neural network models to stress tests for both NLI and QA tasks. Nevertheless, they are still very fragile and demonstrate various unexpected behaviors, thus revealing that there is still room for future improvement in this field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑