理解基于智能体与检索模型的LLM事实核查在不同地理区域间的不平等性
Understanding Inequality of LLM Fact-Checking over Geographic Regions with Agent and Retrieval models
AI总结：
本文评估了LLM在六个全球区域事实核查的不平等性，发现无论采用仅提供声明、Wikipedia智能体还是RAG系统等场景，全球北方声明的表现均显著优于全球南方，凸显了改进数据集平衡与检索策略的迫切需求。
摘要：
事实核查是大型语言模型（LLMs）的一项潜在有用应用，可用于对抗日益增长的虚假信息传播。然而，LLMs的性能在不同地理区域间存在差异。在本文中，我们评估了开源和私有模型在多种不同区域和场景下的事实准确性。使用一个包含600条事实核查声明的数据集，这些声明在六个全球区域间保持平衡，我们考察了核查声明的三种实验设置：（1）仅提供声明可用时，（2）当使用一个具备Wikipedia访问权限的基于LLM的智能体时，以及（3）作为最佳情况场景，当采用一个提供了官方事实核查的检索增强生成（RAG）系统时。我们的发现表明，无论使用何种场景和LLM，包括GPT-4、Claude Sonnet和LLaMA，来自全球北方的声明表现均显著优于来自全球南方的声明。此外，在更切实际的基于Wikipedia智能体的系统案例中，这一差距被进一步扩大，这突显了过于宽泛的知识库在处理特定区域的细微差别方面能力有限。这些结果强调了迫切需要更好的数据集平衡和稳健的检索策略，以增强LLM的事实核查能力，特别是在地理环境多样的背景下。
英文摘要：
Fact-checking is a potentially useful application of Large Language Models (LLMs) to combat the growing dissemination of disinformation. However, the performance of LLMs varies across geographic regions. In this paper, we evaluate the factual accuracy of open and private models across a diverse set of regions and scenarios. Using a dataset containing 600 fact-checked statements balanced across six global regions we examine three experimental setups of fact-checking a statement: (1) when just the statement is available, (2) when an LLM-based agent with Wikipedia access is utilized, and (3) as a best case scenario when a Retrieval-Augmented Generation (RAG) system provided with the official fact check is employed. Our findings reveal that regardless of the scenario and LLM used, including GPT-4, Claude Sonnet, and LLaMA, statements from the Global North perform substantially better than those from the Global South. Furthermore, this gap is broadened for the more realistic case of a Wikipedia agent-based system, highlighting that overly general knowledge bases have a limited ability to address region-specific nuances. These results underscore the urgent need for better dataset balancing and robust retrieval strategies to enhance LLM fact-checking capabilities, particularly in geographically diverse contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
