自信地评估人群
Evaluating the Crowd with Confidence
摘要:
工人质量控制是众包系统的一个关键方面;通常占用了投入众包的很大一部分时间和金钱。在这项工作中,我们设计的技术,以产生工人出错率估计的置信区间,从而使工人质量的更好的评价。我们表明,我们的技术在一系列真实世界的数据集上生成正确的置信区间,并通过使用它们来驱逐表现不佳的工人,并提供答案准确性的置信区间,从而展示了广泛的适用性。
英文摘要:
Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。