arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.04408cs.HCcs.CL
收藏

iSEA:一个用于NLP模型语义错误分析的交互式管道

iSEA: An Interactive Pipeline for Semantic Error Analysis of NLP Models

New York University(纽约大学);Salesforce Research(赛富时研究院)

Jun Yuan, Jesse Vig, Nazneen Rajani

摘要:

NLP模型中的错误分析对于成功的模型开发和部署至关重要。诊断错误的一种常见方法是识别数据集中模型产生最多错误的子群体。然而,现有的方法通常基于预定义的特征来定义子群体,这需要用户提前形成错误的假设。为了补充这些方法,我们提出了iSEA,一个用于NLP模型中语义错误分析的交互式管道,它可以在人机交互系统的上下文中自动发现具有高出错率的语义接地子群体。iSEA使模型开发人员能够通过发现的子群体更多地了解其模型误差,通过对发现的子群体进行交互式分析来验证误差来源,并通过定义自定义子群体来测试有关模型误差的假设。该工具支持在标记和概念级别对易出错的子群体进行语义描述,以及预定义的更高级别的功能。通过用例和专家访谈,我们展示了iSEA如何帮助错误理解和分析。

英文摘要:

Error analysis in NLP models is essential to successful model development and deployment. One common approach for diagnosing errors is to identify subpopulations in the dataset where the model produces the most errors. However, existing approaches typically define subpopulations based on pre-defined features, which requires users to form hypotheses of errors in advance. To complement these approaches, we propose iSEA, an Interactive Pipeline for Semantic Error Analysis in NLP Models, which automatically discovers semantically-grounded subpopulations with high error rates in the context of a human-in-the-loop interactive system. iSEA enables model developers to learn more about their model errors through discovered subpopulations, validate the sources of errors through interactive analysis on the discovered subpopulations, and test hypotheses about model errors by defining custom subpopulations. The tool supports semantic descriptions of error-prone subpopulations at the token and concept level, as well as pre-defined higher-level features. Through use cases and expert interviews, we demonstrate how iSEA can assist error understanding and analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑