arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2504.08697cs.CL
收藏

大语言模型作为跨度标注器:大语言模型与人类的比较研究

LLMs as Span Annotators: A Comparative Study of LLMs and Humans

Charles University(查理大学);ETH Zurich(苏黎世联邦理工学院);Edinburgh Napier University(爱丁堡纳皮尔大学);trivago N.V.(trivago公司);TU Darmstadt, Germany(德累斯顿理工大学)

Zdeněk Kasner, Vilém Zouhar, Patrícia Schmidtová, Ivan Kartáč, Kristýna Onderková, Ondřej Plátek, Dimitra Gkatzia, Saad Mahamood, Ondřej Dušek, Simone Balloccu

AI总结:

本文比较了大语言模型与人类在跨度标注任务中的表现,发现LLMs在成本和准确性上具有竞争力,且提供了大规模标注数据集。

摘要:

跨度标注——在跨度层面标注特定文本特征——可以用于评估那些单评分指标无法提供可行反馈的文本。直到最近,跨度标注都是由人工标注员或微调模型完成的。在本文中,我们研究大语言模型(LLMs)是否能成为人类标注员的替代品。我们比较了LLMs在三个跨度标注任务上的能力:评估数据到文本生成、识别翻译错误以及检测宣传技巧。我们证明,总体而言,LLMs与人类标注员之间的一致性(IAA)仅是中等的。然而,我们展示了LLMs犯错的率与熟练的众包工作者相似。LLMs还以极低的成本产生标注。我们发布了包含超过40,000个模型和人类跨度标注的数据集,供进一步研究。

英文摘要:

Span annotation - annotating specific text features at the span level - can be used to evaluate texts where single-score metrics fail to provide actionable feedback. Until recently, span annotation was done by human annotators or fine-tuned models. In this paper, we study whether large language models (LLMs) can serve as an alternative to human annotators. We compare the abilities of LLMs to skilled human annotators on three span annotation tasks: evaluating data-to-text generation, identifying translation errors, and detecting propaganda techniques. We show that overall, LLMs have only moderate inter-annotator agreement (IAA) with human annotators. However, we demonstrate that LLMs make errors at a similar rate as skilled crowdworkers. LLMs also produce annotations at a fraction of the cost per output annotation. We release the dataset of over 40k model and human span annotations for further research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑