ExplainaBoard：一个可解释的NLP排行榜
ExplainaBoard: An Explainable Leaderboard for NLP
摘要：
随着NLP研究的快速发展，排行榜已经成为跟踪各种系统在各种NLP任务上的性能的一种工具。在某种程度上，它们在这个目标上是有效的，但通常呈现出提交系统的相当简单的一维视图，仅通过整体准确性数字进行传达。在本文中，我们提出了一个新的概念和实现NLP评估：解释板，它除了继承标准排行榜的功能外，还允许研究人员（i）诊断单个系统的优点和缺点（例如，性能最好的系统不擅长什么？）(ii)解释多个系统之间的关系。(e.g.~系统A在哪里优于系统B？如果我们把系统A、B和C结合起来会怎么样？）以及（iii）仔细检查预测结果（例如~多个系统的常见错误是什么，或者在什么情况下会发生特定的错误？到目前为止，ExplainaBoard涵盖了400多个系统，50个数据集，40种语言和12个任务。ExplainaBoard不断更新，最近升级支持（1）多语言多任务基准测试，（2）元评估，（3）更复杂的任务：机器翻译，这也是评论家建议的。我们不仅在网站\url{http://explainaboard.nlpedia.ai/}上发布了一个在线平台，而且还在Github \url{https://github.com/neulab/explainaBoard}和PyPi \url{https://pypi.org/project/interpret-eval/}上使用MIT许可证将我们的评估工具作为API，允许用户方便地离线评估他们的模型。此外，我们还发布了我们运行或收集的系统的所有输出文件，以激励未来的“输出驱动”研究。
英文摘要：
With the rapid development of NLP research, leaderboards have emerged as one tool to track the performance of various systems on various NLP tasks. They are effective in this goal to some extent, but generally present a rather simplistic one-dimensional view of the submitted systems, communicated only through holistic accuracy numbers. In this paper, we present a new conceptualization and implementation of NLP evaluation: the ExplainaBoard, which in addition to inheriting the functionality of the standard leaderboard, also allows researchers to (i) diagnose strengths and weaknesses of a single system (e.g.~what is the best-performing system bad at?) (ii) interpret relationships between multiple systems. (e.g.~where does system A outperform system B? What if we combine systems A, B, and C?) and (iii) examine prediction results closely (e.g.~what are common errors made by multiple systems, or in what contexts do particular errors occur?). So far, ExplainaBoard covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks. ExplainaBoard keeps updated and is recently upgraded by supporting (1) multilingual multi-task benchmark, (2) meta-evaluation, and (3) more complicated task: machine translation, which reviewers also suggested.} We not only released an online platform on the website \url{http://explainaboard.nlpedia.ai/} but also make our evaluation tool an API with MIT Licence at Github \url{https://github.com/neulab/explainaBoard} and PyPi \url{https://pypi.org/project/interpret-eval/} that allows users to conveniently assess their models offline. We additionally release all output files from systems that we have run or collected to motivate "output-driven" research in the future.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
