媒体偏见检测中的定义敏感性：多定义数据集与基准
Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark
浏览论文内容
中文总结 AI 辅助
该研究针对媒体偏见检测中定义差异被忽视的问题，通过含354名参与者的实验和四个LLM评估，发现概念框架影响标注，发布了多定义偏见检测数据集MUDD。
中文摘要 AI 辅助
媒体偏见检测依赖于定义和示例来明确什么属于偏见，但即使名称相同，这些定义在不同数据集中往往存在差异或保持隐含状态。这种差异使得针对同一偏见类别训练的模型是学习到相同的概念还是不同的现象变得不明确，而这一问题在先前研究中大多被忽视。我们通过一项包含354名参与者的被试间实验及对四个大型语言模型（LLM）的并行评估，探究定义选择如何影响偏见标注。参与者和模型使用在概念框架和阐述上存在差异的定义，对六篇新闻文章的四个偏见类别进行评分。在总计8496个人类评分和28800个LLM评分中，我们发现定义的概念目标会导致标注差异，而保留概念的阐述不会：概念框架会显著改变人类的标注，对LLM的影响甚至更强。我们讨论了其对标注协议和基于提示的测量中概念规范的影响，并考虑定义敏感性如何在媒体偏见之外传播到下游分类任务。我们还发布了多定义偏见检测数据集MUDD（Multi-Definition Bias Detection Dataset）。
英文摘要
Media bias detection relies on definitions and examples that specify what counts as bias, yet these specifications often vary across datasets or remain implicit, even when given the same name. Such variation makes it unclear whether models trained for the same bias category learn the same construct or different phenomena, a problem largely overlooked in prior work. We examine how definition choice affects bias annotation in a between-subjects experiment with 354 participants and a parallel evaluation with four LLMs. Participants and models rate six news articles across four bias categories using definitions that vary in conceptual framing and elaboration. Across 8,496 human and 28,800 LLM ratings, we find that the conceptual target of a definition drives annotation divergence, while construct-preserving elaboration does not: conceptual framing significantly shifts annotations for humans and does so even more strongly for LLMs. We discuss implications for construct specification in annotation protocols and prompt-based measurement, and consider how definitional sensitivity may propagate to downstream classification beyond media bias. We also release MUDD, the Multi-Definition Bias Detection Dataset.
发表机构
- Technical University of Munich（慕尼黑工业大学）
- National Institute of Informatics（信息学研究所）
- University of Queensland（昆士兰大学）
机构由 AI 辅助整理，请以论文原文为准。
