arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.07019cs.CLcs.CY
收藏

使用大型语言模型评估阴谋论的影响

Assessing the Impact of Conspiracy Theories Using Large Language Models

Arizona State University(亚利桑那州立大学);University of Washington(华盛顿大学);USC Information Sciences Institute(南加州大学信息科学研究所);ASU Institute for Social Science Research(亚利桑那州立大学社会科学研究所)

Bohan Jiang, Dawei Li, Zhen Tan, Xinyi Zhou, Ashwin Rao, Kristina Lerman, H. Russell Bernard, Huan Liu

AI总结:

本文构建人工标注的阴谋论影响数据集,并设计利用 LLM 多步推理的评估策略,发现证据驱动模式更准确,同时揭示模型存在位置、情绪和冗长性偏差。

摘要:

衡量 CT(阴谋论)的相对影响,对于确定应对措施的优先级和有效分配资源非常重要,尤其是在危机期间。然而,评估 CT 对公众的实际影响面临独特挑战。这不仅需要收集针对 CT 的专门知识,还需要来自社会、心理和文化维度的多元信息。大型语言模型(LLM)最近的进展表明,它们在这一情境下具有潜在用途,这不仅因为其从大规模训练语料中获得的广泛知识,也因为它们可被用于复杂推理。在本研究中,我们构建了包含流行 CT 及其人工标注影响的数据集。随后,我们借鉴人类影响评估过程中的洞见,设计了有针对性的策略,以利用 LLM 执行类似人类的 CT 影响评估。通过严格实验,我们发现:一种采用多步推理、批判性分析更多 CT 相关证据的影响评估模式能够产生准确结果;而大多数 LLM 表现出强烈偏差,例如会赋予提示中较早出现的 CT 更高影响,同时对带有强烈情绪且冗长的 CT 生成准确性较低的影响评估。

英文摘要:

Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challenges. It requires not only the collection of CT-specific knowledge but also diverse information from social, psychological, and cultural dimensions. Recent advancements in large language models (LLMs) suggest their potential utility in this context, not only due to their extensive knowledge from large training corpora but also because they can be harnessed for complex reasoning. In this work, we develop datasets of popular CTs with human-annotated impacts. Borrowing insights from human impact assessment processes, we then design tailored strategies to leverage LLMs for performing human-like CT impact assessments. Through rigorous experiments, we textit{discover that an impact assessment mode using multi-step reasoning to analyze more CT-related evidence critically produces accurate results; and most LLMs demonstrate strong bias, such as assigning higher impacts to CTs presented earlier in the prompt, while generating less accurate impact assessments for emotionally charged and verbose CTs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑