我们信任生成式人工智能：聊天机器人能否有效验证政治信息？
In Generative AI we Trust: Can Chatbots Effectively Verify Political Information?
AI总结：
本研究比较ChatGPT与Bing Chat检测政治信息真实性的能力，发现ChatGPT平均准确率72%优于Bing Chat的67%，且受语言、主题和来源偏见影响，凸显LLM聊天机器人在应对虚假信息方面的潜力与局限。
摘要：
本文对两种基于大型语言模型（LLM）的聊天机器人——ChatGPT和Bing Chat（最近更名为Microsoft Copilot）——在检测政治信息真实性方面的能力进行了比较分析。我们采用人工智能审计方法来调查聊天机器人如何评估关于五个主题的真实、虚假和边缘性陈述：COVID-19、俄罗斯对乌克兰的侵略、大屠杀、气候变化以及涉及LGBTQ+群体的辩论。我们通过使用英语、俄语和乌克兰语的提示，比较了聊天机器人在高资源和低资源语言中的表现。此外，我们探索了聊天机器人根据政治传播学中的虚假信息、错误信息和阴谋论概念来评估陈述的能力，使用了面向定义的提示。我们还系统地测试了这些评估如何受到来源偏见的影响，我们通过将特定主张归因于各种政治和社会行为者来模拟这种偏见。结果显示，ChatGPT在基线真实性评估任务中表现优异，在无需预训练的情况下，跨语言平均正确评估了72%的案例。Bing Chat表现较差，准确率为67%。我们观察到，聊天机器人在评估高资源和低资源语言的提示时存在显著差异，并且它们如何将评估适应于政治传播概念，ChatGPT提供的输出比Bing Chat更为细致。最后，我们发现对于某些真实性检测相关任务，聊天机器人的表现因陈述的主题或归因的来源而异。这些发现凸显了基于LLM的聊天机器人在应对在线环境中不同形式虚假信息方面的潜力，但也指出了由于特定因素（如提示的语言或主题）而导致这种潜力实现方式的显著差异。
英文摘要：
This article presents a comparative analysis of the ability of two large language model (LLM)-based chatbots, ChatGPT and Bing Chat, recently rebranded to Microsoft Copilot, to detect veracity of political information. We use AI auditing methodology to investigate how chatbots evaluate true, false, and borderline statements on five topics: COVID-19, Russian aggression against Ukraine, the Holocaust, climate change, and LGBTQ+ related debates. We compare how the chatbots perform in high- and low-resource languages by using prompts in English, Russian, and Ukrainian. Furthermore, we explore the ability of chatbots to evaluate statements according to political communication concepts of disinformation, misinformation, and conspiracy theory, using definition-oriented prompts. We also systematically test how such evaluations are influenced by source bias which we model by attributing specific claims to various political and social actors. The results show high performance of ChatGPT for the baseline veracity evaluation task, with 72 percent of the cases evaluated correctly on average across languages without pre-training. Bing Chat performed worse with a 67 percent accuracy. We observe significant disparities in how chatbots evaluate prompts in high- and low-resource languages and how they adapt their evaluations to political communication concepts with ChatGPT providing more nuanced outputs than Bing Chat. Finally, we find that for some veracity detection-related tasks, the performance of chatbots varied depending on the topic of the statement or the source to which it is attributed. These findings highlight the potential of LLM-based chatbots in tackling different forms of false information in online environments, but also points to the substantial variation in terms of how such potential is realized due to specific factors, such as language of the prompt or the topic.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
