面向大规模语言模型事实核查的全面阶段性基准评估
Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking
AI总结：
FactArena提出一个全面的LLM事实核查阶段基准评估框架，通过标准化流程和动态生成挑战性声明，揭示LLM在事实推理中的鲁棒性与准确性差异。
摘要：
大规模语言模型（LLMs）正在越来越多地应用于现实中的事实核查系统，但现有的评估主要集中在声明验证上，忽略了更广泛的事实核查流程，包括声明提取和证据检索。这种狭窄的焦点阻碍了当前基准在揭示现代LLMs系统性推理失败、事实盲区和鲁棒性限制方面的能力。为弥合这一差距，我们提出了FactArena，一个完全自动化的竞技场式评估框架，用于对LLMs进行全面、分阶段的基准测试，涵盖完整的事实核查流程。FactArena集成了三个关键组件：（i）一个由LLM驱动的事实核查流程，标准化声明分解、通过工具增强的交互进行证据检索以及基于论证的判断预测；（ii）一个由整合参考指南引导的竞技场式判断机制，确保在异质判断代理之间进行无偏和一致的成对比较；（iii）一个由竞技场驱动的声明演化模块，能够自适应生成更具挑战性和语义可控的声明，以测试LLMs在固定种子数据之外的事实鲁棒性。在16种最先进的LLMs上，FactArena产生了稳定且可解释的排名。我们的分析进一步揭示了静态声明验证准确率与端到端事实核查能力之间的显著差异，突显了全面评估的必要性。所提出的框架提供了一种可扩展且可信的范式，用于诊断LLMs的事实推理能力，指导未来模型开发，并推进LLMs在安全关键的事实核查应用中的可靠部署。
英文摘要：
Large Language Models (LLMs) are increasingly deployed in real-world fact-checking systems, yet existing evaluations focus predominantly on claim verification and overlook the broader fact-checking workflow, including claim extraction and evidence retrieval. This narrow focus prevents current benchmarks from revealing systematic reasoning failures, factual blind spots, and robustness limitations of modern LLMs. To bridge this gap, we present FactArena, a fully automated arena-style evaluation framework that conducts comprehensive, stage-wise benchmarking of LLMs across the complete fact-checking pipeline. FactArena integrates three key components: (i) an LLM-driven fact-checking process that standardizes claim decomposition, evidence retrieval via tool-augmented interactions, and justification-based verdict prediction; (ii) an arena-styled judgment mechanism guided by consolidated reference guidelines to ensure unbiased and consistent pairwise comparisons across heterogeneous judge agents; and (iii) an arena-driven claim-evolution module that adaptively generates more challenging and semantically controlled claims to probe LLMs' factual robustness beyond fixed seed data. Across 16 state-of-the-art LLMs spanning seven model families, FactArena produces stable and interpretable rankings. Our analyses further reveal significant discrepancies between static claim-verification accuracy and end-to-end fact-checking competence, highlighting the necessity of holistic evaluation. The proposed framework offers a scalable and trustworthy paradigm for diagnosing LLMs' factual reasoning, guiding future model development, and advancing the reliable deployment of LLMs in safety-critical fact-checking applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
