arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.01906cs.CLcs.AI
收藏

Dynatask:一个创建动态AI基准测试任务的框架

Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks

University College London(伦敦大学学院);Google(谷歌);Coactive AI(Coactive人工智能公司);Hugging Face(Hugging Face公司);Facebook AI Research(脸书人工智能研究院);The University of Hong Kong(香港大学)

Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, Douwe Kiela

摘要:

我们介绍Dynatask:一个用于设置自定义NLP任务的开源系统,旨在大大降低托管和评估最先进的NLP模型所需的技术知识和工作量,以及与众包工作者进行循环数据汇集所需的模型。Dynatask与Dynabench集成,Dynabench是一个研究平台,用于重新思考人工智能中的基准测试,促进人类和模型的循环数据汇集和评估。要创建一个任务,用户只需要编写一个简短的任务配置文件,相关的Web界面和模型托管基础设施将自动生成。该系统可在https://dynabench.org/上获得,完整的库可在https://github.com/facebookresearch/dynabench上找到。

英文摘要:

We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers. Dynatask is integrated with Dynabench, a research platform for rethinking benchmarking in AI that facilitates human and model in the loop data collection and evaluation. To create a task, users only need to write a short task configuration file from which the relevant web interfaces and model hosting infrastructure are automatically generated. The system is available at https://dynabench.org/ and the full library can be found at https://github.com/facebookresearch/dynabench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑