arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09467cs.CL
收藏

让人类留在回路中:基于生成式AI的以人为中心的自动化标注

Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

University of Pennsylvania(宾夕法尼亚大学)

Nicholas Pangakis, Samuel Wolken

AI总结:

本文提出以人为中心的框架评估自动化标注,使用GPT-4在11个数据集上复制27项任务,发现LLM表现存在显著差异,强调人工验证标签对负责任评估至关重要。

摘要:

自动化文本标注是生成式大型语言模型(LLMs)在社交媒体研究中的一个引人注目的用例。近期研究表明,LLMs在标注任务上能取得出色表现;然而,这些研究仅在少量任务上评估LLMs,且由于依赖公开基准数据集,可能受到数据污染。在此,我们测试了一个以人为中心的框架,用于负责任地评估自动化标注中使用的人工智能工具。我们使用GPT-4,基于来自高影响力期刊上近期发表的计算机社会科学论文中的11个受密码保护的数据集,复制了27项标注任务。对于每项任务,我们将GPT-4的标注结果与人工标注的真实标签,以及在人工生成标签上微调的独立监督分类模型的标注结果进行比较。尽管LLM标签的质量普遍较高,但我们发现LLM在不同任务上的表现存在显著差异,甚至在同一数据集内也是如此。我们的研究结果强调了以人为中心的工作流程和严格评估标准的重要性:尽管采用了提示微调等多种优化策略,在众多场景中,自动化标注与人类判断仍存在显著分歧。将自动化标注建立在人类生成的验证标签基础之上,对于负责任的评估至关重要。

英文摘要:

Automated text annotation is a compelling use case for generative large language models (LLMs) in social media research. Recent work suggests that LLMs can achieve strong performance on annotation tasks; however, these studies evaluate LLMs on a small number of tasks and likely suffer from contamination due to a reliance on public benchmark datasets. Here, we test a human-centered framework for responsibly evaluating artificial intelligence tools used in automated annotation. We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets from recently published computational social science articles in high-impact journals. For each task, we compare GPT-4 annotations against human-annotated ground-truth labels and against annotations from separate supervised classification models fine-tuned on human-generated labels. Although the quality of LLM labels is generally high, we find significant variation in LLM performance across tasks, even within datasets. Our findings underscore the importance of a human-centered workflow and careful evaluation standards: Automated annotations significantly diverge from human judgment in numerous scenarios, despite various optimization strategies such as prompt tuning. Grounding automated annotation in validation labels generated by humans is essential for responsible evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑