arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.05154cs.AIcs.HC
收藏

Mephisto:一个可移植、可复制和迭代的众包框架

Mephisto: A Framework for Portable, Reproducible, and Iterative Crowdsourcing

Meta AI

Jack Urbanek, Pratik Ringshia

摘要:

我们介绍Mephisto,这是一个使研究众包更具可复制性,透明性和协作性的框架。Mephisto提供了涵盖广泛的任务设计和数据汇集工作流的抽象,并提供了一个简单的用户体验,使最佳实践易于默认。在本白皮书中,我们讨论了机器学习研究中数据汇集和注释的现状,建立了构建共享框架的动机,使研究人员能够创建和源数据汇集和注释工具,作为其出版物的一部分,并概述了一组建议的系统要求,以促进这些目标。然后,我们逐步介绍Mephisto中的解决方案,解释我们使用的抽象,我们围绕用户体验的设计决策,并分享实现细节以及它们与原始动机的一致性。我们还讨论了当前的局限性,以及未来继续实现框架的初始目标的工作。Mephisto是一个开源项目,其文档可以在www.mephisto.ai上找到。

英文摘要:

We introduce Mephisto, a framework to make crowdsourcing for research more reproducible, transparent, and collaborative. Mephisto provides abstractions that cover a broad set of task designs and data collection workflows, and provides a simple user experience to make best-practices easy defaults. In this whitepaper we discuss the current state of data collection and annotation in ML research, establish the motivation for building a shared framework to enable researchers to create and open-source data collection and annotation tools as part of their publication, and outline a set of suggested requirements for a system to facilitate these goals. We then step through our resolution in Mephisto, explaining the abstractions we use, our design decisions around the user experience, and share implementation details and where they align with the original motivations. We also discuss current limitations, as well as future work towards continuing to deliver on the framework's initial goals. Mephisto is available as an open source project, and its documentation can be found at www.mephisto.ai.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑