LLM标注在数据集标注中的前景与陷阱：以媒体偏见检测为例
The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection
AI总结：
本研究探讨LLM自动标注在媒体偏见检测中的可行性，构建含48000余样本的数据集annolexical，微调分类器超越标注LLM且接近人工标注性能，显著降低成本并揭示局限。
摘要：
通过雇佣或众包进行标注的高昂成本，使得创建用于训练可靠文本分类器的大规模高质量数据集变得复杂。近期研究建议使用大型语言模型（LLMs）来自动化标注过程，在保持数据质量的同时降低这些成本。LLMs在仇恨言论检测和政治框架标注等下游任务中已展现出有前景的结果。基于这些领域的成功，本研究探讨了LLMs是否适用于标注媒体偏见检测这一复杂任务，以及是否可以在此类数据上训练下游的媒体偏见分类器。我们创建了annolexical，这是首个用于媒体偏见分类的大规模数据集，包含超过48000个合成标注的样本。我们在此数据集上微调的分类器，在Matthews相关系数（MCC）上超越了所有标注LLMs 5-9个百分点，并且在两个媒体偏见基准数据集（BABE和BASIL）上评估时，其性能接近或优于在人工标注数据上训练的模型。本研究展示了我们的方法如何显著降低媒体偏见领域数据集创建的成本，进而降低分类器开发的成本，而我们后续的行为压力测试则揭示了其当前的一些局限性和权衡。
英文摘要：
High annotation costs from hiring or crowdsourcing complicate the creation of large, high-quality datasets needed for training reliable text classifiers. Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. LLMs have shown promising results in annotating downstream tasks like hate speech detection and political framing. Building on the success in these areas, this study investigates whether LLMs are viable for annotating the complex task of media bias detection and whether a downstream media bias classifier can be trained on such data. We create annolexical, the first large-scale dataset for media bias classification with over 48000 synthetically annotated examples. Our classifier, fine-tuned on this dataset, surpasses all of the annotator LLMs by 5-9 percent in Matthews Correlation Coefficient (MCC) and performs close to or outperforms the model trained on human-labeled data when evaluated on two media bias benchmark datasets (BABE and BASIL). This study demonstrates how our approach significantly reduces the cost of dataset creation in the media bias domain and, by extension, the development of classifiers, while our subsequent behavioral stress-testing reveals some of its current limitations and trade-offs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
