运行受控众包实验的挑战和策略
Challenges and strategies for running controlled crowdsourcing experiments
摘要：
本文报告了我们在众包平台中进行受控实验时遇到的挑战和学到的经验教训。众包正在成为一种有吸引力的技术，可以在实验研究中吸引多样化和大量的受试者，使研究人员能够实现在实验室环境中不可行的规模和完成时间。然而，这种规模和灵活性是以多种有时未知的偏见和混杂因素为代价的，这些偏见和混杂因素来自众包平台的技术限制和在“野外”进行受控实验的挑战。在本文中，我们把我们的经验，在运行系统的评估任务设计作为一个激励的例子，探索，描述和量化的潜在影响，运行不受控制的众包实验，并得出可能的应对策略。在确定的挑战中，我们可以提到抽样偏差，控制受试者的实验条件，学习效果和众包结果的可靠性。根据我们的实证研究，潜在的偏见和混杂因素的影响可以达到38%的损失，在不受控制的设置中收集的数据的效用;它可以显着改变实验的结果。这些问题最终促使我们实施了CrowdHub，这是一个位于主要众包平台之上的系统，允许研究人员和从业者运行受控的众包项目。
英文摘要：
This paper reports on the challenges and lessons we learned while running controlled experiments in crowdsourcing platforms. Crowdsourcing is becoming an attractive technique to engage a diverse and large pool of subjects in experimental research, allowing researchers to achieve levels of scale and completion times that would otherwise not be feasible in lab settings. However, the scale and flexibility comes at the cost of multiple and sometimes unknown sources of bias and confounding factors that arise from technical limitations of crowdsourcing platforms and from the challenges of running controlled experiments in the "wild". In this paper, we take our experience in running systematic evaluations of task design as a motivating example to explore, describe, and quantify the potential impact of running uncontrolled crowdsourcing experiments and derive possible coping strategies. Among the challenges identified, we can mention sampling bias, controlling the assignment of subjects to experimental conditions, learning effects, and reliability of crowdsourcing results. According to our empirical studies, the impact of potential biases and confounding factors can amount to a 38\% loss in the utility of the data collected in uncontrolled settings; and it can significantly change the outcome of experiments. These issues ultimately inspired us to implement CrowdHub, a system that sits on top of major crowdsourcing platforms and allows researchers and practitioners to run controlled crowdsourcing projects.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
