稳健健身房：统一NLP评估环境
Robustness Gym: Unifying the NLP Evaluation Landscape
摘要：
尽管在标准基准测试中表现出色，但深度神经网络在部署到现实世界的系统中时往往很脆弱。因此，最近的研究集中在测试这些模型的鲁棒性上，从而产生了从对抗攻击到基于规则的数据转换的各种评估方法。在这项工作中，我们确定了评估NLP系统的挑战，并提出了一个解决方案，鲁棒性健身房（RG）的形式，一个简单的和可扩展的评估工具包，统一了4个标准的评估范式：子群，转换，评估集，和对抗攻击。通过提供一个通用的评估平台，Robustness Gym使从业者只需点击几下就可以比较所有4种评估范式的结果，并使用内置的一组抽象轻松开发和共享新的评估方法。为了验证Robustness Gym对从业者的实用性，我们与情感建模团队进行了一个真实的案例研究，揭示了18%以上的性能下降。为了验证健壮健身房可以帮助新的研究分析，我们进行了第一次研究的国家的最先进的商业和学术命名实体链接（NEL）系统，以及细粒度分析的国家的最先进的摘要模型。对于NEL来说，商业系统很难链接稀有实体，并落后于学术同行10%以上，而最先进的摘要模型在需要抽象和蒸馏的示例上很难，降低了9%以上。鲁棒性健身房可以在https://robustnessgym.com/上找到
英文摘要：
Despite impressive performance on standard benchmarks, deep neural networks are often brittle when deployed in real-world systems. Consequently, recent research has focused on testing the robustness of such models, resulting in a diverse set of evaluation methodologies ranging from adversarial attacks to rule-based data transformations. In this work, we identify challenges with evaluating NLP systems and propose a solution in the form of Robustness Gym (RG), a simple and extensible evaluation toolkit that unifies 4 standard evaluation paradigms: subpopulations, transformations, evaluation sets, and adversarial attacks. By providing a common platform for evaluation, Robustness Gym enables practitioners to compare results from all 4 evaluation paradigms with just a few clicks, and to easily develop and share novel evaluation methods using a built-in set of abstractions. To validate Robustness Gym's utility to practitioners, we conducted a real-world case study with a sentiment-modeling team, revealing performance degradations of 18%+. To verify that Robustness Gym can aid novel research analyses, we perform the first study of state-of-the-art commercial and academic named entity linking (NEL) systems, as well as a fine-grained analysis of state-of-the-art summarization models. For NEL, commercial systems struggle to link rare entities and lag their academic counterparts by 10%+, while state-of-the-art summarization models struggle on examples that require abstraction and distillation, degrading by 9%+. Robustness Gym can be found at https://robustnessgym.com/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
