arXiv 2501.025322025-01-05cs.CLcs.AIcs.CY

在潜在内容分析中对照人类标注者评估大型语言模型：情感、政治倾向、情绪强度与讽刺

Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm

Institute for Artificial Intelligence Research and Development of Serbia（塞尔维亚人工智能研究与发展研究所）；University of Belgrade（贝尔格莱德大学）；Institute for Philosophy and Social Theory（哲学与社会理论研究所）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau (RPTU)（莱茵兰-普法尔茨州 Kaiserslautern-Landau 应用科技大学）；German Research Centre for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Vilnius Gediminas Technical University（维尔纽斯格迪米纳斯技术大学）；Drexel University（德雷塞尔大学）；University of Montenegro（黑山大学）；Faculty of Dramatic Arts（戏剧艺术学院）；University of Kragujevac（克拉古耶瓦茨大学）；Faculty of Engineering（工程学院）

Ljubisa Bojic, Olga Zagovora, Asta Zelenkauskaite, Vuk Vukovic, Milan Cabarkapa, Selma Veseljević Jerkovic, Ana Jovančevic

备注：: 24 pages, 3 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过33名人类标注者与八个 LLM 变体对100条文本的多任务标注，比较情感、政治倾向、情绪强度和讽刺检测的可靠性，发现 GPT-4 等模型在部分任务上可稳定复现人类判断。

摘要：

在数字通信快速发展的时代，每天都会产生大量文本数据，因此需要高效的潜在内容分析方法以提取有意义的洞见。大型语言模型（LLM）为实现这一过程的自动化提供了可能，但目前缺乏在多个维度上将其表现与人类标注者进行比较的全面评估。本研究评估了七个最先进 LLM（包括 OpenAI 的 GPT-4、Gemini、Llama 和 Mixtral 的变体）相对于人类标注者在分析情感、政治倾向、情绪强度和讽刺检测方面的可靠性、一致性与质量。共有33名人类标注者和八个 LLM 变体评估了100条精选文本项目，产生3,300条人类标注和19,200条 LLM 标注；LLM 在三个时间点接受评估，以考察时间一致性。研究者使用 Krippendorff's alpha 测量评分者间信度，并使用组内相关系数评估随时间变化的一致性。结果显示，人类和 LLM 在情感分析与政治倾向评估中均表现出较高可靠性，且 LLM 的内部一致性高于人类。在情绪强度方面，LLM 表现出比人类更高的一致性，不过人类给出的情绪强度评分显著更高。两组在讽刺检测上都存在困难，低一致性即为证明。LLM 在所有维度上均表现出优异的时间一致性，表明其表现随时间保持稳定。本研究得出结论：LLM（尤其是 GPT-4）能够在情感和政治倾向上有效复现人类分析，但人类专业知识对于情绪强度解释仍然必不可少。研究结果证明了 LLM 在潜在内容分析的某些领域实现一致且高质量表现的潜力。

英文摘要：

In the era of rapid digital communication, vast amounts of textual data are generated daily, demanding efficient methods for latent content analysis to extract meaningful insights. Large Language Models (LLMs) offer potential for automating this process, yet comprehensive assessments comparing their performance to human annotators across multiple dimensions are lacking. This study evaluates the reliability, consistency, and quality of seven state-of-the-art LLMs, including variants of OpenAI's GPT-4, Gemini, Llama, and Mixtral, relative to human annotators in analyzing sentiment, political leaning, emotional intensity, and sarcasm detection. A total of 33 human annotators and eight LLM variants assessed 100 curated textual items, generating 3,300 human and 19,200 LLM annotations, with LLMs evaluated across three time points to examine temporal consistency. Inter-rater reliability was measured using Krippendorff's alpha, and intra-class correlation coefficients assessed consistency over time. The results reveal that both humans and LLMs exhibit high reliability in sentiment analysis and political leaning assessments, with LLMs demonstrating higher internal consistency than humans. In emotional intensity, LLMs displayed higher agreement compared to humans, though humans rated emotional intensity significantly higher. Both groups struggled with sarcasm detection, evidenced by low agreement. LLMs showed excellent temporal consistency across all dimensions, indicating stable performance over time. This research concludes that LLMs, especially GPT-4, can effectively replicate human analysis in sentiment and political leaning, although human expertise remains essential for emotional intensity interpretation. The findings demonstrate the potential of LLMs for consistent and high-quality performance in certain areas of latent content analysis.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。