arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.04118cs.CLcs.LG
收藏

超越精度:使用CheckList对NLP模型进行行为测试

Beyond Accuracy: Behavioral Testing of NLP models with CheckList

Microsoft Research(微软研究院);Univ. of Washington(华盛顿大学);Univ. of California, Irvine(加州大学欧文分校)

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh

摘要:

虽然测量保持精度一直是评估泛化的主要方法,但它往往高估了NLP模型的性能,而评估模型的替代方法要么关注单个任务,要么关注特定行为。受软件工程中行为测试原理的启发,我们引入了CheckList,一种用于测试NLP模型的任务不可知方法。CheckList包括一个通用语言能力和测试类型的矩阵,可以促进全面的测试构思,以及一个软件工具,可以快速生成大量不同数量的测试用例。我们说明了实用的CheckList与测试的三个任务,确定在商业和国家的最先进的模型中的关键故障。在一项用户研究中,一个负责商业情感分析模型的团队在一个经过广泛测试的模型中发现了新的可操作的错误。在另一项用户研究中,使用CheckList的NLP从业者创建了两倍多的测试,发现的bug几乎是没有它的用户的三倍。

英文摘要:

Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model. In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑