Dynabench：重新思考NLP中的标杆管理
Dynabench: Rethinking Benchmarking in NLP
摘要：
我们介绍Dynabench，一个用于动态数据集创建和模型基准测试的开源平台。Dynabench在Web浏览器中运行，并支持人和模型在环数据集创建：注释者试图创建目标模型会错误分类但另一个人不会的示例。在本文中，我们认为Dynabench解决了我们社区的一个关键需求：当代模型在基准任务上迅速实现了出色的性能，但在简单的挑战示例中失败，并在现实世界的场景中摇摇欲坠。使用Dynabench，数据集创建、模型开发和模型评估可以直接相互通知，从而产生更强大和信息量更大的基准。我们报告了四个初始NLP任务，说明了这些概念并强调了平台的承诺，并解决了动态基准测试作为该领域新标准的潜在反对意见。
英文摘要：
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
