质量保证：重新思考影像AI中的标注策略
Quality Assured: Rethinking Annotation Strategies in Imaging AI
AI总结：
本文研究了标注公司内部质量保证（QA）对影像AI数据标注质量的影响，基于57,648张实例分割图像发现标注公司优于MTurk，且改进标注说明比投资内部QA更能提升标注效率。
摘要：
本文未描述一种新方法，而是研究了基于AI的图像分析在可靠基准测试及最终实际应用中的一个重要基础：生成高质量参考标注。以往研究侧重于将众包作为外包标注的手段，但鲜少关注标注公司，特别是其内部质量保证（QA）流程。因此，我们的目标是评估标注公司采用的QA对标注质量的影响，并设计最大化数据标注效率的方法。基于从四家标注公司及Amazon Mechanical Turk（MTurk）共924名标注员和34名QA工人处获得的57,648张实例分割图像，我们得出以下见解：（1）与广泛使用的MTurk平台相比，标注公司在数量和质量上均表现更佳。（2）标注公司的内部QA仅提供边际改进，甚至没有改进。然而，改进标注说明而非投资QA可以大幅提升标注性能。（3）内部QA的益处取决于特定的图像特征。我们的工作能够使研究人员从固定的标注预算中获取更多价值，并改变标注公司开展内部QA的方式。
英文摘要：
This paper does not describe a novel method. Instead, it studies an essential foundation for reliable benchmarking and ultimately real-world application of AI-based image analysis: generating high-quality reference annotations. Previous research has focused on crowdsourcing as a means of outsourcing annotations. However, little attention has so far been given to annotation companies, specifically regarding their internal quality assurance (QA) processes. Therefore, our aim is to evaluate the influence of QA employed by annotation companies on annotation quality and devise methodologies for maximizing data annotation efficacy. Based on a total of 57,648 instance segmented images obtained from a total of 924 annotators and 34 QA workers from four annotation companies and Amazon Mechanical Turk (MTurk), we derived the following insights: (1) Annotation companies perform better both in terms of quantity and quality compared to the widely used platform MTurk. (2) Annotation companies' internal QA only provides marginal improvements, if any. However, improving labeling instructions instead of investing in QA can substantially boost annotation performance. (3) The benefit of internal QA depends on specific image characteristics. Our work could enable researchers to derive substantially more value from a fixed annotation budget and change the way annotation companies conduct internal QA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
