使用Crowd-Kit从众包中学习
Learning from Crowds with Crowd-Kit
AI总结:
本文提出通用众包质量控制工具包Crowd-Kit,集成真值推断、众包深度学习和数据质量估计等方法,支持多模态答案与基准测试,并以Apache 2.0协议开源。
摘要:
本文介绍了Crowd-Kit,一个用于众包的通用计算质量控制工具包。Crowd-Kit在Python中提供了流行质量控制算法的高效便捷实现,包括真值推断、从众包中进行深度学习以及数据质量估计等方法。我们的工具包支持多种答案模态,并提供数据集加载器和示例notebook,以便更快地进行原型开发。我们在多个不同性质的数据集上对该工具包进行了广泛评估,从而能够使用相同的代码库以统一、系统化和可复现的方式对计算质量控制方法进行基准测试。我们在Apache License 2.0下于https://github.com/Toloka/crowd-kit发布我们的代码和数据。
英文摘要:
This paper presents Crowd-Kit, a general-purpose computational quality control toolkit for crowdsourcing. Crowd-Kit provides efficient and convenient implementations of popular quality control algorithms in Python, including methods for truth inference, deep learning from crowds, and data quality estimation. Our toolkit supports multiple modalities of answers and provides dataset loaders and example notebooks for faster prototyping. We extensively evaluated our toolkit on several datasets of different natures, enabling benchmarking computational quality control methods in a uniform, systematic, and reproducible way using the same codebase. We release our code and data under the Apache License 2.0 at https://github.com/Toloka/crowd-kit.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。