新闻验证器大比拼：ChatGPT 3.5、ChatGPT 4.0、Bing AI 和 Bard 在新闻事实核查中的比较性能评估
News Verifiers Showdown: A Comparative Performance Evaluation of ChatGPT 3.5, ChatGPT 4.0, Bing AI, and Bard in News Fact-Checking
AI总结：
本研究通过黑盒测试评估了ChatGPT 3.5、ChatGPT 4.0、Bing AI和Bard在新闻事实核查中的表现，发现GPT-4.0以71分领先，但所有AI模型在理解新闻语境上仍不及人类，凸显了持续提升AI能力及保留人类核查的必要性。
摘要：
本研究旨在评估知名大型语言模型（LLMs），即 OpenAI 的 ChatGPT 3.5 和 4.0、Google 的 Bard（LaMDA）以及 Microsoft 的 Bing AI 在使用黑盒测试辨别新闻条目真实性方面的能力。共有 100 条经过事实核查的新闻条目，全部来自独立事实核查机构，在受控条件下被呈现给每个 LLM。它们的回复被分为三类：True（真）、False（假）和 Partially True/False（部分真/假）。LLMs 的有效性是根据其分类结果与独立机构提供的已验证事实的准确性来衡量的。结果显示，所有模型均表现出中等水平的能力，平均得分为 100 分中的 65.25 分。在这些模型中，OpenAI 的 GPT-4.0 以 71 分脱颖而出，表明较新的 LLM 在区分事实与欺骗方面具有优势。然而，与人类事实核查员的表现相比，AI 模型尽管前景广阔，但在理解新闻信息中固有的细微差别和语境方面仍显不足。研究结果突显了 AI 在事实核查领域的潜力，同时强调了人类认知技能的持续重要性以及不断提升 AI 能力的必要性。最后，本工作模拟产生的实验数据已在 Kaggle 上公开提供。
英文摘要：
This study aimed to evaluate the proficiency of prominent Large Language Models (LLMs), namely OpenAI's ChatGPT 3.5 and 4.0, Google's Bard(LaMDA), and Microsoft's Bing AI in discerning the truthfulness of news items using black box testing. A total of 100 fact-checked news items, all sourced from independent fact-checking agencies, were presented to each of these LLMs under controlled conditions. Their responses were classified into one of three categories: True, False, and Partially True/False. The effectiveness of the LLMs was gauged based on the accuracy of their classifications against the verified facts provided by the independent agencies. The results showed a moderate proficiency across all models, with an average score of 65.25 out of 100. Among the models, OpenAI's GPT-4.0 stood out with a score of 71, suggesting an edge in newer LLMs' abilities to differentiate fact from deception. However, when juxtaposed against the performance of human fact-checkers, the AI models, despite showing promise, lag in comprehending the subtleties and contexts inherent in news information. The findings highlight the potential of AI in the domain of fact-checking while underscoring the continued importance of human cognitive skills and the necessity for persistent advancements in AI capabilities. Finally, the experimental data produced from the simulation of this work is openly available on Kaggle.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
