ERASER：一个评价并行化NLP模型的基准
ERASER: A Benchmark to Evaluate Rationalized NLP Models
摘要：
NLP中最先进的模型现在主要基于深度神经网络，这些网络在如何进行预测方面是不透明的。这一限制增加了人们对为NLP设计更多可解释的深度模型的兴趣，这些模型揭示了模型输出背后的“推理”。但这方面的工作是在不同的数据集和任务上进行的，目标和指标也相应独特;这使得跟踪进展变得困难。我们提出了评估语义和简单英语推理（ERASER）基准，以推进NLP中可解释模型的研究。该基准包括多个数据集和任务，其中已收集了人类对“基本原理”（支持证据）的注释。我们提出了几个指标，旨在捕捉模型提供的基本原理与人类基本原理的一致程度，以及这些基本原理的忠实程度（即，所提供的基本原理影响相应预测的程度）。我们希望发布这个基准可以促进设计更可解释的NLP系统的进展。基准测试、代码和文档可在https://www.eraserbenchmark.com/上获得
英文摘要：
State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP that reveal the `reasoning' behind model outputs. But work in this direction has been conducted on different datasets and tasks with correspondingly unique aims and metrics; this makes it difficult to track progress. We propose the Evaluating Rationales And Simple English Reasoning (ERASER) benchmark to advance research on interpretable models in NLP. This benchmark comprises multiple datasets and tasks for which human annotations of "rationales" (supporting evidence) have been collected. We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i.e., the degree to which provided rationales influenced the corresponding predictions). Our hope is that releasing this benchmark facilitates progress on designing more interpretable NLP systems. The benchmark, code, and documentation are available at https://www.eraserbenchmark.com/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
