建模和减轻人为注释错误，以设计具有人机回路机器学习的高效流处理系统
Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning
摘要：
高质量的人工注释对于创建有效的机器学习驱动的流处理系统是必要的。我们研究混合流处理系统的基础上，一个人在回路机器学习（HITL-ML）的范例，其中一个或多个人类注释和自动分类器（训练至少部分由人类注释）标签传入流的实例。这是许多近实时社交媒体分析和网络应用程序的典型特征，包括数字志愿者团体在紧急情况下注释社交媒体帖子。从实践的角度来看，低质量的人类注释会导致错误的标签，用于重新训练自动分类器，并间接有助于创建不准确的分类器。将人类注释视为一种心理过程，使我们能够解决这些局限性。我们表明，人类注释质量取决于注释者所示的实例的顺序，并且可以通过提供给注释者的实例序列/顺序的局部变化来改善，从而产生更准确的流注释。我们为人类注释任务调整了错误和失误的理论动机的人为错误框架，以研究排序实例的影响（即，“注释时间表”）。此外，我们提出了一个错误避免方法的主动学习范例流处理应用程序的鲁棒性，这些可能的人为错误（在滑的形式）时，决定一个人的注释时间表。我们使用众包实验支持人为错误框架，并通过在自然灾害期间过滤相关社交媒体帖子的分类任务的广泛实验，针对主动学习的标准基线评估所提出的算法。
英文摘要：
High-quality human annotations are necessary for creating effective machine learning-driven stream processing systems. We study hybrid stream processing systems based on a Human-In-The-Loop Machine Learning (HITL-ML) paradigm, in which one or many human annotators and an automatic classifier (trained at least partially by the human annotators) label an incoming stream of instances. This is typical of many near-real-time social media analytics and web applications, including annotating social media posts during emergencies by digital volunteer groups. From a practical perspective, low-quality human annotations result in wrong labels for retraining automated classifiers and indirectly contribute to the creation of inaccurate classifiers. Considering human annotation as a psychological process allows us to address these limitations. We show that human annotation quality is dependent on the ordering of instances shown to annotators and can be improved by local changes in the instance sequence/order provided to the annotators, yielding a more accurate annotation of the stream. We adapt a theoretically-motivated human error framework of mistakes and slips for the human annotation task to study the effect of ordering instances (i.e., an "annotation schedule"). Further, we propose an error-avoidance approach to the active learning paradigm for stream processing applications robust to these likely human errors (in the form of slips) when deciding a human annotation schedule. We support the human error framework using crowdsourcing experiments and evaluate the proposed algorithm against standard baselines for active learning via extensive experimentation on classification tasks of filtering relevant social media posts during natural disasters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
