arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.08482cs.AI
收藏

从多个标签中找到真值:为什么任务参数很重要

Finding the Ground-Truth from Multiple Labellers: Why Parameters of the Task Matter

Caspian(里海公司);Wolfson Childhood Cancer Research Centre(沃尔夫森儿童癌症研究中心)

Robert McCluskey, Amir Enshaei, Bashar Awwad Shiekh Hasan

摘要:

近年来,雇用多个工人来标记机器学习模型的数据变得越来越重要,因为人们越来越需要收集大量的标记数据来训练复杂模型,同时减轻不正确和嘈杂的标记风险。无论是在流行的众包平台上进行大规模数据收集,还是在高专业知识的标签实践中进行较小规模的工人,都有各种方法可以推荐,以收集就业工人的共识并建立真值标签。然而,很少有关于标签任务的各种参数如何影响所述方法的研究。这些参数包括工人数量、工人专业知识、分类中的标签数量和样本大小。在本文中,多数票,CrowdTruth和二项式期望最大化的研究对这些参数的排列,以提供更好的理解的参数设置,以提供优势,在真值推断。研究结果表明,期望最大化和CrowdTruth都只可能在某些参数条件下比多数投票更具优势,而在许多情况下,这些方法可以被证明没有重大影响。给出了指导,以什么参数下的方法最好的工作,而实验框架提供了一种方法来测试其他已建立的方法,并测试新的方法,可以尝试提供有利的性能,本文中的方法没有。还对最佳众包参数有了更深入的了解。

英文摘要:

Employing multiple workers to label data for machine learning models has become increasingly important in recent years with greater demand to collect huge volumes of labelled data to train complex models while mitigating the risk of incorrect and noisy labelling. Whether it is large scale data gathering on popular crowd-sourcing platforms or smaller sets of workers in high-expertise labelling exercises, there are various methods recommended to gather a consensus from employed workers and establish ground-truth labels. However, there is very little research on how the various parameters of a labelling task can impact said methods. These parameters include the number of workers, worker expertise, number of labels in a taxonomy and sample size. In this paper, Majority Vote, CrowdTruth and Binomial Expectation Maximisation are investigated against the permutations of these parameters in order to provide better understanding of the parameter settings to give an advantage in ground-truth inference. Findings show that both Expectation Maximisation and CrowdTruth are only likely to give an advantage over majority vote under certain parameter conditions, while there are many cases where the methods can be shown to have no major impact. Guidance is given as to what parameters methods work best under, while the experimental framework provides a way of testing other established methods and also testing new methods that can attempt to provide advantageous performance where the methods in this paper did not. A greater level of understanding regarding optimal crowd-sourcing parameters is also achieved.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑