arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.05286cs.LGcs.AIcs.HC
收藏

面向人类数据标注的可泛化错误建模:来自工业级搜索数据标注项目的证据

Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program

Apple(苹果公司)

Heinrich Peters, Alireza Hashemi, James Rae

AI总结:

本文提出一种基于任务特征与行为特征的预测性错误模型,用于检测工业级搜索数据标注任务中的潜在错误,实验证明其具有良好泛化能力并能显著提升标注审计效率与质量。

摘要:

机器学习(ML)和人工智能(AI)系统在训练和评估过程中严重依赖人类标注的数据。在此背景下,一个主要挑战是标注错误的出现,因为其影响会降低模型性能。本文提出了一种预测性错误模型,该模型经过训练,旨在检测三个工业级ML应用(音乐流媒体、视频流媒体和移动应用)的搜索相关性标注任务中的潜在错误。基于来自一项广泛搜索相关性标注项目的真实数据,我们证明错误可以通过中等水平的模型性能(AUC=0.65-0.75)进行预测,并且模型性能在不同应用之间具有良好的泛化能力(即一个全局的、任务无关的模型表现与特定任务模型相当)。与以往通常侧重于从特定任务特征预测标注标签的研究不同,我们的模型经过训练,可以直接从任务特征和源自标注过程的行为特征的组合中预测错误,以实现高度的泛化能力。我们展示了该模型在审计场景中的实用性,在该场景中,优先处理具有高预测错误概率的任务会显著增加纠正后的标注错误数量(例如,音乐流媒体应用的效率提升了40%)。这些结果强调,行为错误检测模型可以在数据标注过程的效率和质量方面产生显著改善。我们的发现揭示了对数据标注过程中有效错误管理的关键见解,从而为更广泛的人类在环ML领域做出了贡献。

英文摘要:

Machine learning (ML) and artificial intelligence (AI) systems rely heavily on human-annotated data for training and evaluation. A major challenge in this context is the occurrence of annotation errors, as their effects can degrade model performance. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industry-scale ML applications (music streaming, video streaming, and mobile apps). Drawing on real-world data from an extensive search relevance annotation program, we demonstrate that errors can be predicted with moderate model performance (AUC=0.65-0.75) and that model performance generalizes well across applications (i.e., a global, task-agnostic model performs on par with task-specific models). In contrast to past research, which has often focused on predicting annotation labels from task-specific features, our model is trained to predict errors directly from a combination of task features and behavioral features derived from the annotation process, in order to achieve a high degree of generalizability. We demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities considerably increases the amount of corrected annotation errors (e.g., 40% efficiency gains for the music streaming application). These results highlight that behavioral error detection models can yield considerable improvements in the efficiency and quality of data annotation processes. Our findings reveal critical insights into effective error management in the data annotation process, thereby contributing to the broader field of human-in-the-loop ML.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑