众包中微任务间资格测试的有效性研究
Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing
摘要：
众包中的资格测试通常用于通过测量员工执行微任务的能力来预过滤员工。虽然为每种任务类型创建资格测试被认为是一种常见且合理的方法，但本研究调查了当在多种任务类型中使用相同的资格测试时，其员工过滤性能。在Amazon Mechanical Turk上，我们在六种不同的情况下测试了注释准确性，其中任务由两个不同的难度级别组成，来自相同的现实世界领域：四种组合情况，其中资格测试和实际任务彼此相同或不同，以及另外两种情况，即要求具有硕士资格的工人只执行实际任务。实验结果证明了以下两个发现：i）被分配到困难的资格测试的工人得分更高，无论实际任务的难度如何; ii）具有硕士资格的工人在低难度任务中获得了更好的注释准确性，但在高难度任务中不如通过资格测试的人准确。
英文摘要：
Qualification tests in crowdsourcing are often used to pre-filter workers by measuring their ability in executing microtasks.While creating qualification tests for each task type is considered as a common and reasonable way, this study investigates into its worker-filtering performance when the same qualification test is used across multiple types of tasks.On Amazon Mechanical Turk, we tested the annotation accuracy in six different cases where tasks consisted of two different difficulty levels, arising from the identical real-world domain: four combinatory cases in which the qualification test and the actual task were the same or different from each other, as well as two other cases where workers with Masters Qualification were asked to perform the actual task only.The experimental results demonstrated the two following findings: i) Workers that were assigned to a difficult qualification test scored better annotation accuracy regardless of the difficulty of the actual task; ii) Workers with Masters Qualification scored better annotation accuracy on the low-difficulty task, but were not as accurate as those who passed a qualification test on the high-difficulty task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
