百万富翁：一种提示引导的众包方法
Millionaire: A Hint-guided Approach for Crowdsourcing
摘要：
现代机器学习正在向复杂模型时代迁移，这需要大量注释良好的数据。虽然众包是实现这一目标的一个有前途的工具，但现有的众包方法几乎无法获得足够数量的高质量标签。本文以《百万富翁》游戏节目中的“带提示的猜测”回答策略为启发，将提示引导的方法引入众包中以应对这一挑战。我们的方法鼓励员工在不确定问题时从提示中获得帮助。具体来说，我们提出了一个混合阶段设置，包括主阶段和提示阶段。当工作人员在主舞台上面对任何不确定的问题时，他们被允许进入提示阶段，在做出任何回答之前查找提示。一个独特的支付机制，满足两个重要的设计原则的众包。此外，建议的机制进一步鼓励高素质的工人少使用提示，这有助于识别和分配更大的可能性支付给他们。在Amazon Mechanical Turk上进行的实验表明，该方法能够以较低的成本确保足够数量的高质量标签，并检测出高质量的工人。
英文摘要：
Modern machine learning is migrating to the era of complex models, which requires a plethora of well-annotated data. While crowdsourcing is a promising tool to achieve this goal, existing crowdsourcing approaches barely acquire a sufficient amount of high-quality labels. In this paper, motivated by the "Guess-with-Hints" answer strategy from the Millionaire game show, we introduce the hint-guided approach into crowdsourcing to deal with this challenge. Our approach encourages workers to get help from hints when they are unsure of questions. Specifically, we propose a hybrid-stage setting, consisting of the main stage and the hint stage. When workers face any uncertain question on the main stage, they are allowed to enter the hint stage and look up hints before making any answer. A unique payment mechanism that meets two important design principles for crowdsourcing is developed. Besides, the proposed mechanism further encourages high-quality workers less using hints, which helps identify and assigns larger possible payment to them. Experiments are performed on Amazon Mechanical Turk, which show that our approach ensures a sufficient number of high-quality labels with low expenditure and detects high-quality workers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
