事实核查AI生成的新闻报道：LLM能发现自己的谎言吗？
Fact-checking AI-generated news reports: Can LLMs catch their own lies?
AI总结：
本研究评估LLM核查自身及其他LLM生成新闻声明真实性的能力，发现其核查效果受新闻类型、信息属性等影响，引入RAG可减少无法评估的情况但会增加错误，指出需优化检索质量，动态事件与地方新闻或需人机协同核查。
摘要：
本文评估了大语言模型（Large Language Models，LLM）评估由自身或其他LLM生成的“新闻报道”中声明真实性的能力。我们的目标是确定LLM能否使用与核查人类声明类似的方法，有效对自身生成的内容进行事实核查。研究结果表明，LLM评估国内或国际新闻报道中声明的效果优于地方新闻报道，评估静态信息的效果优于动态信息，验证真实声明的效果优于虚假声明。我们假设，这种差异的产生是因为前几类声明在训练数据中的呈现更充分。此外，我们发现在检索增强生成（Retrieval-Augmented Generation，RAG）框架中引入搜索引擎的检索结果，能显著减少LLM无法评估的声明数量。但这种方法也会增加错误评估的发生，部分原因是检索结果存在无关或质量低下的问题。这项诊断性研究强调，未来针对机器生成报道的事实核查研究需优先提升检索信息的精准度与相关性，以更好地支撑事实核查工作。此外，关于动态事件和地方新闻的声明可能需要人机协同的事实核查系统，以确保准确性和可靠性。
英文摘要：
In this paper, we evaluate the ability of Large Language Models (LLMs) to assess the veracity of claims in ''news reports'' generated by themselves or other LLMs. Our goal is to determine whether LLMs can effectively fact-check their own content, using methods similar to those used to verify claims made by humans. Our findings indicate that LLMs are more effective at assessing claims in national or international news stories than in local news stories, better at evaluating static information than dynamic information, and better at verifying true claims compared to false ones. We hypothesize that this disparity arises because the former types of claims are better represented in the training data. Additionally, we find that incorporating retrieved results from a search engine in a Retrieval-Augmented Generation (RAG) setting significantly reduces the number of claims an LLM cannot assess. However, this approach also increases the occurrence of incorrect assessments, partly due to irrelevant or low-quality search results. This diagnostic study highlights the need for future research on fact-checking machine-generated reports to prioritize improving the precision and relevance of retrieved information to better support fact-checking efforts. Furthermore, claims about dynamic events and local news may require human-in-the-loop fact-checking systems to ensure accuracy and reliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
