LabelAId：众包系统中用于提高人工标注质量和领域知识的即时AI干预
LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems
AI总结：
本文提出LabelAId，结合程序化弱监督与FT-Transformer的即时AI干预模型，提升众包标注质量与领域知识，实验显示错误推断准确率提升36.7%，并在Project Sidewalk中验证了其有效性。
摘要：
众包平台已经改变了分布式问题解决的方式，但质量控制仍然是一个持续的挑战。传统的质量控制措施，如预筛选工作者和完善指令，往往只专注于优化经济产出。本文探讨了即时AI干预，以增强众包工作者的标注质量和领域特定知识。我们引入了LabelAId，一种结合程序化弱监督（PWS）和FT-Transformer的高级推理模型，基于用户行为和领域知识推断标注正确性。我们的技术评估表明，LabelAId流水线始终优于最先进的机器学习基线，在50个下游样本下将错误推断准确率提高了36.7%。随后，我们将LabelAId实施到Project Sidewalk中，这是一个用于城市无障碍的开源众包平台。一项涉及34名参与者的受试者间研究表明，LabelAId显著提高了标注精度，同时不损害效率，并增加了标注者的信心。我们讨论了LabelAId的成功因素、局限性及其在其他众包科学领域的通用性。
英文摘要：
Crowdsourcing platforms have transformed distributed problem-solving, yet quality control remains a persistent challenge. Traditional quality control measures, such as prescreening workers and refining instructions, often focus solely on optimizing economic output. This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness based on user behavior and domain knowledge. Our technical evaluation shows that our LabelAId pipeline consistently outperforms state-of-the-art ML baselines, improving mistake inference accuracy by 36.7% with 50 downstream samples. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility. A between-subjects study with 34 participants demonstrates that LabelAId significantly enhances label precision without compromising efficiency while also increasing labeler confidence. We discuss LabelAId's success factors, limitations, and its generalizability to other crowdsourced science domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
