arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1602.00575cs.LG
收藏

通过具有多个选项的众包进行多对象分类

Multi-object Classification via Crowdsourcing with a Reject Option

Syracuse University(雪城大学);IBM Thomas J. Watson Research Center(IBM托马斯·J·沃森研究中心);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Qunwei Li, Aditya Vempaty, Lav R. Varshney, Pramod K. Varshney

摘要:

考虑为$M$元分类任务设计一个有效的众包系统。群组工作者完成简单的二进制微任务,其结果被聚合以给出最终结果。我们考虑了一个新的场景,其中工作人员有一个拒绝选项,这样当他们无法或选择不响应时,他们可以跳过微任务。例如,在不匹配的语音转录中,不懂语言的工人可能无法对专注于其分类感知之外的语音维度的微任务做出反应。我们提出了一种聚合方法,使用加权多数表决规则,每个工人的响应被分配一个优化的权重,以最大限度地提高人群的分类性能。我们评估系统的性能在精确和渐近形式。此外,我们还考虑了这样一种设置,即可能有一组贪婪的工作者,即使他们无法可靠地执行微任务,他们也会完成微任务。我们考虑了一个不经意的和一个清除策略来处理贪婪的工人,开发一个算法来适应性地在这两个之间切换的基础上估计的分数贪婪的工人在匿名人群。仿真结果表明,改进的性能相比,传统的多数表决。

英文摘要:

Consider designing an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final result. We consider the novel scenario where workers have a reject option so they may skip microtasks when they are unable or choose not to respond. For example, in mismatched speech transcription, workers who do not know the language may not be able to respond to microtasks focused on phonological dimensions outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize the crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Further, we consider the setting where there may be a set of greedy workers that complete microtasks even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy to deal with greedy workers, developing an algorithm to adaptively switch between the two based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show improved performance compared with conventional majority voting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑