当多数表决失败时:比较嘈杂的人类计算环境的质量保证方法
When majority voting fails: Comparing quality assurance methods for noisy human computation environment
摘要:
质量保证仍然是人类计算研究的一个关键主题。先前的工作表明,多数表决是有效的低难度的任务,但有困难的任务的限制。本文探讨了解决这个问题的两种方法:锦标赛选择和淘汰选择,利用2,3和4路不同的答案之间的比较,人类计算任务。我们的实验结果和统计分析表明,这两种方法产生正确的答案在嘈杂的人类计算环境中更经常比多数表决。此外,我们发现,相对于使用双向比较,使用4向比较可以显着降低质量保证的成本。
英文摘要:
Quality assurance remains a key topic in human computation research. Prior work indicates that majority voting is effective for low difficulty tasks, but has limitations for harder tasks. This paper explores two methods of addressing this problem: tournament selection and elimination selection, which exploit 2-, 3- and 4-way comparisons between different answers to human computation tasks. Our experimental results and statistical analyses show that both methods produce the correct answer in noisy human computation environment more often than majority voting. Furthermore, we find that the use of 4-way comparisons can significantly reduce the cost of quality assurance relative to the use of 2-way comparisons.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。