将工人视角纳入NLP的MTurk标注实践
Incorporating Worker Perspectives into MTurk Annotation Practices for NLP
AI总结：
本研究通过文献综述和工人调查，揭示MTurk工人偏好与NLP研究者认知的差异，并提出改进标注实践以尊重工人权利和提升数据质量的建议。
摘要：
当前在Amazon Mechanical Turk (MTurk)上为自然语言处理收集数据的实践，通常依赖于关于数据质量的研究和NLP研究人员之间共享的启发式方法的结合。然而，如果不考虑MTurk工人的视角，这些方法容易受到关于工人权利和响应质量不佳问题的影响。我们进行了一项批判性文献综述和一项针对MTurk工人的调查，旨在解决关于公平报酬、工人隐私、数据质量以及考虑工人激励的最佳实践的开放性问题。我们发现，工人的偏好往往与NLP研究人员中的普遍认知相悖。受访工人更倾向于可靠、合理的报酬，而非不确定的、非常高的报酬；他们报告经常在人口统计问题上撒谎；并对工作被无理由拒绝表示沮丧。我们还发现，工人认为一些质量控制方法，如要求最短响应时间或Master资格，是有偏见且基本无效的。基于调查结果，我们提供了关于未来NLP研究如何更好地考虑MTurk工人经验，以尊重工人权利并提高数据质量的建议。
英文摘要：
Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics shared among NLP researchers. However, without considering the perspectives of MTurk workers, these approaches are susceptible to issues regarding workers' rights and poor response quality. We conducted a critical literature review and a survey of MTurk workers aimed at addressing open questions regarding best practices for fair payment, worker privacy, data quality, and considering worker incentives. We found that worker preferences are often at odds with received wisdom among NLP researchers. Surveyed workers preferred reliable, reasonable payments over uncertain, very high payments; reported frequently lying on demographic questions; and expressed frustration at having work rejected with no explanation. We also found that workers view some quality control methods, such as requiring minimum response times or Master's qualifications, as biased and largely ineffective. Based on the survey results, we provide recommendations on how future NLP studies may better account for MTurk workers' experiences in order to respect workers' rights and improve data quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
