协作式人机风险标注:使用CHAIRA共同标注网络不文明行为
Collaborative Human-AI Risk Annotation: Co-Annotating Online Incivility with CHAIRA
AI总结:
本文提出CHAIRA,一种协作式人机AI风险标注工具,利用LLMs和四种提示策略支持人类与AI共同标注网络不文明行为,实验表明最佳协作提示能实现接近双人编码者的一致性,并揭示了AI与人类在标注中的互补性与挑战。
摘要:
协作式人机AI标注是处理大规模和复杂数据的各种任务的一种有前景的方法。支持有效人机AI协作进行数据标注的工具和方法是一个重要的研究方向。在本文中,我们提出了CHAIRA:一种协作式人机AI风险标注工具,使人类和AI智能体能够协作标注网络不文明行为。我们利用大型语言模型(LLMs)来促进人类和AI标注者之间的交互,并检验了四种不同的提示策略。所开发的CHAIRA系统将多种提示方法与人类和AI的协作相结合,用于网络不文明行为数据标注。我们在457条带有真实标签的用户评论上对CHAIRA进行了评估,该评估基于人类和AI编码者之间的评分者间一致性。我们发现,最具协作性的提示策略支持人类智能体和AI之间的高度一致性,与两名人类编码者之间的一致性相当。虽然AI遗漏了一些人类编码者容易识别的隐式不文明行为,但它也发现了人类编码者忽视的政治上微妙的不文明行为。我们的研究揭示了使用AI智能体进行不文明行为标注的益处和挑战,并为主观数据标注中的人机AI协作提供了设计启示和最佳实践。
英文摘要:
Collaborative human-AI annotation is a promising approach for various tasks with large-scale and complex data. Tools and methods to support effective human-AI collaboration for data annotation are an important direction for research. In this paper, we present CHAIRA: a Collaborative Human-AI Risk Annotation tool that enables human and AI agents to collaboratively annotate online incivility. We leveraged Large Language Models (LLMs) to facilitate the interaction between human and AI annotators and examine four different prompting strategies. The developed CHAIRA system combines multiple prompting approaches with human-AI collaboration for online incivility data annotation. We evaluated CHAIRA on 457 user comments with ground truth labels based on the inter-rater agreement between human and AI coders. We found that the most collaborative prompt supported a high level of agreement between a human agent and AI, comparable to that of two human coders. While the AI missed some implicit incivility that human coders easily identified, it also spotted politically nuanced incivility that human coders overlooked. Our study reveals the benefits and challenges of using AI agents for incivility annotation and provides design implications and best practices for human-AI collaboration in subjective data annotation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。