基于深度学习的文本分类算法对实际输入扰动的敏感性研究
On Sensitivity of Deep Learning Based Text Classification Algorithms to Practical Input Perturbations
摘要:
文本分类是一项基本的自然语言处理任务,具有广泛的应用,其中深度学习方法已经产生了最先进的结果。虽然这些模型因其黑盒性质而受到严厉批评,但它们对输入文本中轻微扰动的鲁棒性一直是一个值得关注的问题。在这项工作中,我们进行了一项以数据为中心的研究,评估系统实际扰动对基于深度学习的文本分类模型(如CNN、LSTM和基于BERT的算法)性能的影响。扰动是由添加和删除不需要的标记(如标点符号和停止词)引起的,这些标记与模型的最终性能关系最小。我们证明了这些深度学习方法,包括BERT,对四个标准基准数据集SST 2,TREC-6,BBC News和tweet_eval的合法输入扰动是敏感的。我们观察到,与添加令牌相比,BERT更容易受到令牌移除的影响。此外,与基于CNN的模型相比,LSTM对输入扰动稍微更敏感。这项工作也作为一个实用的指南,以评估列车测试条件的差异对模型的最终性能的影响。
英文摘要:
Text classification is a fundamental Natural Language Processing task that has a wide variety of applications, where deep learning approaches have produced state-of-the-art results. While these models have been heavily criticized for their black-box nature, their robustness to slight perturbations in input text has been a matter of concern. In this work, we carry out a data-focused study evaluating the impact of systematic practical perturbations on the performance of the deep learning based text classification models like CNN, LSTM, and BERT-based algorithms. The perturbations are induced by the addition and removal of unwanted tokens like punctuation and stop-words that are minimally associated with the final performance of the model. We show that these deep learning approaches including BERT are sensitive to such legitimate input perturbations on four standard benchmark datasets SST2, TREC-6, BBC News, and tweet_eval. We observe that BERT is more susceptible to the removal of tokens as compared to the addition of tokens. Moreover, LSTM is slightly more sensitive to input perturbations as compared to CNN based model. The work also serves as a practical guide to assessing the impact of discrepancies in train-test conditions on the final performance of models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。