SYNTHEVAL:使用合成检查清单对NLP模型进行混合行为测试
SYNTHEVAL: Hybrid Behavioral Testing of NLP Models with Synthetic CheckLists
AI总结:
提出SYNTHEVAL混合行为测试框架,利用LLM生成测试类型并识别挑战性例子,结合人工模板设计,有效评估NLP模型在情感分析和有毒语言检测中的弱点。
摘要:
传统的NLP基准测试通常使用静态的留出测试集。然而,这种方法往往导致对性能的高估,并且缺乏对NLP模型进行全面、可解释和动态评估的能力。最近,像DynaBench(Kiela等人,2021)和CheckList(Ribeiro等人,2020)这样的工作通过使用由多步骤人工标注流程生成的测试类型对NLP模型进行行为测试,解决了这些局限性。不幸的是,手动创建各种测试类型需要大量人力,通常成本高昂。在这项工作中,我们提出了SYNTHEVAL,一个混合行为测试框架,利用大型语言模型(LLMs)生成广泛的测试类型,以对NLP模型进行全面评估。SYNTHEVAL首先通过LLMs使用受控生成来生成句子,然后通过比较LLMs做出的预测与特定任务NLP模型的预测来识别具有挑战性的例子。在最后阶段,人类专家调查这些具有挑战性的例子,手动设计模板,并识别特定任务模型持续表现出的失败类型。我们将SYNTHEVAL应用于两个分类任务:情感分析和有毒语言检测,并表明我们的框架在识别这些任务中强模型的弱点方面是有效的。我们在https://github.com/Loreley99/SynthEval_CheckList分享了我们的代码。
英文摘要:
Traditional benchmarking in NLP typically involves using static held-out test sets. However, this approach often results in an overestimation of performance and lacks the ability to offer comprehensive, interpretable, and dynamic assessments of NLP models. Recently, works like DynaBench (Kiela et al., 2021) and CheckList (Ribeiro et al., 2020) have addressed these limitations through behavioral testing of NLP models with test types generated by a multistep human-annotated pipeline. Unfortunately, manually creating a variety of test types requires much human labor, often at prohibitive cost. In this work, we propose SYNTHEVAL, a hybrid behavioral testing framework that leverages large language models (LLMs) to generate a wide range of test types for a comprehensive evaluation of NLP models. SYNTHEVAL first generates sentences via LLMs using controlled generation, and then identifies challenging examples by comparing the predictions made by LLMs with task-specific NLP models. In the last stage, human experts investigate the challenging examples, manually design templates, and identify the types of failures the taskspecific models consistently exhibit. We apply SYNTHEVAL to two classification tasks, sentiment analysis and toxic language detection, and show that our framework is effective in identifying weaknesses of strong models on these tasks. We share our code in https://github.com/Loreley99/SynthEval_CheckList.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。