用于宣传检测的混合标注：将 LLM 预标注与人类智能相结合
Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence
AI总结：
提出融合 LLM 预标注与人工验证的宣传检测混合标注框架，通过层级分类、LLM 数据蒸馏训练 SLM，显著提升标注一致性与效率。
摘要：
由于任务复杂且高质量标注数据有限，社交媒体上的宣传检测仍然具有挑战性。本文介绍了一种新颖框架，将人类专业知识与大型语言模型（LLM）辅助相结合，以提升标注一致性和可扩展性。我们提出了一种层级分类体系，将14种细粒度宣传技巧组织为三个更宽泛的类别；在 HQP 数据集上开展人工标注研究，结果揭示细粒度标签的标注者间一致性较低；并实现了一条 LLM 辅助的预标注流程，用于抽取宣传性片段、生成简洁解释，以及分配局部标签和一个全局标签。一项二次人工验证研究表明，一致性和时间效率均有显著改善。在此基础上，我们微调较小语言模型（SLM）以执行结构化标注。我们没有基于人工标注进行微调，而是使用高质量的 LLM 生成数据进行训练，使大模型能够生成这些标注，并让较小模型通过知识蒸馏学习生成它们。我们的工作有助于开发可扩展且稳健的宣传检测系统，支持与可持续发展目标16（SDG 16）相一致的透明、问责的媒体生态系统理念。代码已在我们的 GitHub 仓库公开发布。
英文摘要：
Propaganda detection on social media remains challenging due to task complexity and limited high-quality labeled data. This paper introduces a novel framework that combines human expertise with Large Language Model (LLM) assistance to improve both annotation consistency and scalability. We propose a hierarchical taxonomy that organizes 14 fine-grained propaganda techniques into three broader categories, conduct a human annotation study on the HQP dataset that reveals low inter-annotator agreement for fine-grained labels, and implement an LLM-assisted pre-annotation pipeline that extracts propagandistic spans, generates concise explanations, and assigns local labels as well as a global label. A secondary human verification study shows significant improvements in both agreement and time-efficiency. Building on this, we fine-tune smaller language models (SLMs) to perform structured annotation. Instead of fine-tuning on human annotations, we train on high-quality LLM-generated data, allowing a large model to produce these annotations and a smaller model to learn to generate them via knowledge distillation. Our work contributes towards the development of scalable and robust propaganda detection systems, supporting the idea of transparent and accountable media ecosystems in line with SDG 16. The code is publicly available at our GitHub repository.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
