群体与偏好:没有金本位的群体标签的不可能性定理
Crowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard
摘要:
众包的一个常见用途是获取数据集的标签。已经提出了几种算法来识别人群中没有信息的成员,这样他们的标签就可以被忽略,支付他们的费用也可以避免。这些算法的一个共同动机是尝试在没有任何初始可信标记数据集的情况下进行操作。我们将这类算法作为博弈论环境中的机制进行分析,以了解它们为工人创造的激励机制。我们发现一个不可能的结果,没有任何地面真相,当工人有机会共同分享的“偏见”,他们同意,但不提供信息的真正的标签,总是有均衡,所有代理人报告的偏见。少量的金标准数据被发现是足以排除这些平衡。
英文摘要:
A common use of crowd sourcing is to obtain labels for a dataset. Several algorithms have been proposed to identify uninformative members of the crowd so that their labels can be disregarded and the cost of paying them avoided. One common motivation of these algorithms is to try and do without any initial set of trusted labeled data. We analyse this class of algorithms as mechanisms in a game-theoretic setting to understand the incentives they create for workers. We find an impossibility result that without any ground truth, and when workers have access to commonly shared 'prejudices' upon which they agree but are not informative of true labels, there is always equilibria where all agents report the prejudice. A small amount amount of gold standard data is found to be sufficient to rule out these equilibria.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。