通过标签聚合对众包与大语言模型标注质量的比较研究
A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation
AI总结：
本文构建众包数据集基准，比较个体众包与 LLM 标签及聚合标签质量，并提出 Crowd-LLM 混合聚合方法，证明加入优质 LLM 标签可提升众包聚合标签质量。
摘要：
大语言模型（LLM）能否在数据标注任务上超越众包，近来正引起关注。一些工作通过收集新数据集，在某些特定 NLP 任务上，以个体众包工作者和 LLM 工作者的平均表现验证了这一问题。然而，一方面，现有用于众包标注质量研究的数据集尚未被用于此类评估，而这些数据集有可能从不同视角提供可靠评估。另一方面，这些聚合标签的质量至关重要，因为在使用众包时，由同一实例的多个众包标签聚合得到的估计标签才是最终收集的标签。因此，本文首先考察哪些现有众包数据集可用于比较研究，并构建一个基准。随后，我们比较个体众包标签与 LLM 标签的质量，并对聚合标签进行评估。此外，我们提出一种 Crowd-LLM 混合标签聚合方法，并验证其性能。我们发现，将来自优秀 LLM 的 LLM 标签加入现有众包数据集，可以提升这些数据集聚合标签的质量，而且该质量也高于 LLM 标签自身的质量。
英文摘要：
Whether Large Language Models (LLMs) can outperform crowdsourcing on the data annotation task is attracting interest recently. Some works verified this issue with the average performance of individual crowd workers and LLM workers on some specific NLP tasks by collecting new datasets. However, on the one hand, existing datasets for the studies of annotation quality in crowdsourcing are not yet utilized in such evaluations, which potentially provide reliable evaluations from a different viewpoint. On the other hand, the quality of these aggregated labels is crucial because, when utilizing crowdsourcing, the estimated labels aggregated from multiple crowd labels to the same instances are the eventually collected labels. Therefore, in this paper, we first investigate which existing crowdsourcing datasets can be used for a comparative study and create a benchmark. We then compare the quality between individual crowd labels and LLM labels and make the evaluations on the aggregated labels. In addition, we propose a Crowd-LLM hybrid label aggregation method and verify the performance. We find that adding LLM labels from good LLMs to existing crowdsourcing datasets can enhance the quality of the aggregated labels of the datasets, which is also higher than the quality of LLM labels themselves.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
