TestAug：一个增强基于能力的NLP测试的框架
TestAug: A Framework for Augmenting Capability-based NLP Tests
摘要：
最近提出的基于能力的NLP测试允许模型开发人员测试NLP模型的功能能力，揭示传统heldout机制无法检测到的功能故障。然而，现有的工作基于能力的测试需要大量的手工工作和领域的专业知识，在创建测试用例。在本文中，我们研究了一种低成本的方法，利用GPT-3引擎的测试用例生成。我们进一步建议使用分类器来移除GPT-3的无效输出，并将输出扩展为模板以生成更多的测试用例。实验结果表明，TestAug与现有的行为测试方法相比，具有以下三个优点：（1）TestAug可以发现更多的错误;（2）TestAug中的测试用例更加多样化;（3）TestAug大大节省了创建测试套件的人工工作。TestAug的代码和数据可以在我们的项目网站（https：//guanqun-yang.github.io/testaug/）和GitHub（https：//github.com/guanqun-yang/testaug）上找到。
英文摘要：
The recently proposed capability-based NLP testing allows model developers to test the functional capabilities of NLP models, revealing functional failures that cannot be detected by the traditional heldout mechanism. However, existing work on capability-based testing requires extensive manual efforts and domain expertise in creating the test cases. In this paper, we investigate a low-cost approach for the test case generation by leveraging the GPT-3 engine. We further propose to use a classifier to remove the invalid outputs from GPT-3 and expand the outputs into templates to generate more test cases. Our experiments show that TestAug has three advantages over the existing work on behavioral testing: (1) TestAug can find more bugs than existing work; (2) The test cases in TestAug are more diverse; and (3) TestAug largely saves the manual efforts in creating the test suites. The code and data for TestAug can be found at our project website (https://guanqun-yang.github.io/testaug/) and GitHub (https://github.com/guanqun-yang/testaug).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
