流行分类模型对随机和目标性破坏的鲁棒性基准测试
Benchmarking Popular Classification Models' Robustness to Random and Targeted Corruptions
摘要：
文本分类模型，特别是基于神经网络的模型，在许多流行的基准数据集上已经达到了非常高的精度。然而，这样的模型在部署到现实世界的应用程序中时，往往表现不佳。主要原因是这些模型没有针对足够的真实世界自然数据进行测试。根据应用程序用户的不同，模型输入的词汇和风格可能会有很大的不同。这强调了对模型不可知测试数据集的需求，该数据集由自然出现在野外的各种损坏组成。在这样的基准数据集上训练和测试的模型将对真实世界的数据更加强大。然而，这类数据集并不容易获得。在这项工作中，我们解决了这个问题，通过扩展基准数据集沿着自然发生的腐败，如拼写错误，文本噪音和同义词，并使它们公开。通过广泛的实验，我们比较随机和有针对性的腐败策略，使用本地可解释模型不可知推理（LIME）。我们报告了两个流行的文本分类模型中的漏洞，并发现在大多数情况下，有针对性的腐败可以比随机选择更好地暴露模型的漏洞。
英文摘要：
Text classification models, especially neural networks based models, have reached very high accuracy on many popular benchmark datasets. Yet, such models when deployed in real world applications, tend to perform badly. The primary reason is that these models are not tested against sufficient real world natural data. Based on the application users, the vocabulary and the style of the model's input may greatly vary. This emphasizes the need for a model agnostic test dataset, which consists of various corruptions that are natural to appear in the wild. Models trained and tested on such benchmark datasets, will be more robust against real world data. However, such data sets are not easily available. In this work, we address this problem, by extending the benchmark datasets along naturally occurring corruptions such as Spelling Errors, Text Noise and Synonyms and making them publicly available. Through extensive experiments, we compare random and targeted corruption strategies using Local Interpretable Model-Agnostic Explanations(LIME). We report the vulnerabilities in two popular text classification models along these corruptions and also find that targeted corruptions can expose vulnerabilities of a model better than random choices in most cases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
