扩展真相：AI事实核查中的信心悖论
Scaling Truth: The Confidence Paradox in AI Fact-Checking
AI总结：
本研究系统评估九个大型语言模型在47种语言、5000条声明上的事实核查表现，发现小模型高信心低准确率、大模型高准确率低信心的邓宁-克鲁格式悖论，且非英语和全球南方声明差距最大，为多语言基准和政策提供依据。
摘要：
错误信息的泛滥凸显了对可扩展且可靠的事实核查解决方案的需求。大型语言模型（LLMs）在自动化事实核查方面展现出潜力，但其在全球不同语境下的有效性仍不确定。我们使用此前由174个专业事实核查组织在47种语言中评估的5,000条声明，系统评估了九个成熟的大型语言模型，涵盖多个类别（开源/闭源、多种规模、多样架构、基于推理的模型）。我们的方法测试了模型在训练截止日期之后的声明上的泛化能力，并采用了四种提示策略，模拟公民和专业事实核查员的交互方式，以超过240,000条人工标注作为真实基准。研究结果揭示了一种类似于邓宁-克鲁格效应的令人担忧的模式：较小且易于获取的模型表现出高信心但准确性较低，而较大模型则准确性较高但信心较低。这可能导致信息核查中的系统性偏见，因为资源受限的组织通常使用较小的模型。性能差距在非英语语言和源自全球南方的声明中最为显著，这有可能加剧现有的信息不平等。这些结果为未来研究建立了一个多语言基准，并为旨在确保公平获取可信赖的AI辅助事实核查服务的政策提供了证据基础。
英文摘要：
The rise of misinformation underscores the need for scalable and reliable fact-checking solutions. Large language models (LLMs) hold promise in automating fact verification, yet their effectiveness across global contexts remains uncertain. We systematically evaluate nine established LLMs across multiple categories (open/closed-source, multiple sizes, diverse architectures, reasoning-based) using 5,000 claims previously assessed by 174 professional fact-checking organizations across 47 languages. Our methodology tests model generalizability on claims postdating training cutoffs and four prompting strategies mirroring both citizen and professional fact-checker interactions, with over 240,000 human annotations as ground truth. Findings reveal a concerning pattern resembling the Dunning-Kruger effect: smaller, accessible models show high confidence despite lower accuracy, while larger models demonstrate higher accuracy but lower confidence. This risks systemic bias in information verification, as resource-constrained organizations typically use smaller models. Performance gaps are most pronounced for non-English languages and claims originating from the Global South, threatening to widen existing information inequalities. These results establish a multilingual benchmark for future research and provide an evidence base for policy aimed at ensuring equitable access to trustworthy, AI-assisted fact-checking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
