弥合设计AI驱动的灾害管理分析系统中数据标注界面的知识差距
Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems
AI总结：
针对灾害相关Twitter消息分类中专家与初学者标注者的分歧，研究识别其知识与线索成因，并提出Context界面，通过隐藏情境辅助初学者发现错误，实验表明其相较Highlight和Reasoning界面在态度与行为上存在显著差异。
摘要：
数据标注界面主要利用真实标签来引导标注者给出准确回答。随着人工智能（AI）在特定领域专业任务中的应用日益广泛，帮助初学者标注者识别其早期阶段知识如何导致不准确答案变得愈发重要，这进而有助于确保大规模标注的质量。为研究这一问题，我们开展了一项形成性研究，参与者为来自灾害管理领域的8名人员，各自具备不同水平的专业知识。研究目标是理解在对与灾害相关的Twitter消息进行分类时，导致标注者之间分歧的普遍因素，并分析他们各自的回答。我们的分析识别出专家与初学者标注者之间产生分歧的两个主要原因：1）缺乏情境知识或对情况存在不确定性；2）缺少视觉或补充线索。基于这些发现，我们设计了一个Context界面，该界面会生成辅助内容，帮助初学者识别潜在错误，并提供所呈现推文的隐藏情境。总结性研究将Context设计与数据标注UI中两种广泛使用的设计——Highlight界面和基于Reasoning的界面——进行了比较。我们发现这些设计在态度数据和行为数据方面存在显著差异。最后，我们总结了对未来界面设计的启示，这些界面旨在弥合标注者之间的知识差距。
英文摘要：
Data annotation interfaces predominantly leverage ground truth labels to guide annotators toward accurate responses. With the growing adoption of Artificial Intelligence (AI) in domain-specific professional tasks, it has become increasingly important to help beginning annotators identify how their early-stage knowledge can lead to inaccurate answers, which in turn, helps to ensure quality annotations at scale. To investigate this issue, we conducted a formative study involving eight individuals from the field of disaster management, each possessing varying levels of expertise. The goal was to understand the prevalent factors contributing to disagreements among annotators when classifying Twitter messages related to disasters and to analyze their respective responses. Our analysis identified two primary causes of disagreement between expert and beginner annotators: 1) a lack of contextual knowledge or uncertainty about the situation, and 2) the absence of visual or supplementary cues. Based on these findings, we designed a Context interface, which generates aids that help beginners identify potential mistakes and provide the hidden context of the presented tweet. The summative study compares Context design with two widely used designs in data annotation UI, Highlight and Reasoning-based interfaces. We found significant differences between these designs in terms of attitudinal and behavioral data. We conclude with implications for designing future interfaces aiming at closing the knowledge gap among annotators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
