Low-quality results have been a long-standing problem on microtask crowdsourcing platforms, driving away requesters and justifying low wages for workers. To date, workers have been blamed for low-quality results: they are said to make as little effort as possible, do not pay attention to detail, and lack expertise. In this paper, we hypothesize that requesters may also be responsible for low-quality work: they launch unclear task designs that confuse even earnest workers, under-specify edge cases, and neglect to include examples. We introduce prototype tasks, a crowdsourcing strategy requiring all new task designs to launch a small number of sample tasks. Workers attempt these tasks and leave feedback, enabling the re- quester to iterate on the design before publishing it. We report a field experiment in which tasks that underwent prototype task iteration produced higher-quality work results than the original task designs. With this research, we suggest that a simple and rapid iteration cycle can improve crowd work, and we provide empirical evidence that requester "quality" directly impacts result quality.
Enhanced Nearest Neighbor Classification for Crowdsourcing
Purdue University(普渡大学);Binghamton University, State University of New York(纽约州立大学宾汉姆顿分校);University of California, Los Angeles(加州大学洛杉矶分校);Moody’s Analytics, Inc.(穆迪分析公司)
Communicating Uncertainty in AI-Supported Nurse Scheduling: Expert Co-Design and Evaluation of an Explainable AI Prototype
XITASO GmbH(XITASO有限公司);Technische Hochschule Ingolstadt(因戈尔施塔特应用技术大学);University College Cork(科克大学学院);University of Bayreuth(拜罗伊特大学);Augsburg University Hospital(奥格斯堡大学医院)