DeepFact: 深度研究事实性共演基准与智能体
DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality
AI总结:
本文提出DeepFact-Bench基准和DeepFact-Eval智能体,通过审计-评分机制提升事实性验证的可靠性,并在深度研究报告中实现高准确性。
摘要:
Search-augmented LLM agents可以生成深度研究报告(DRRs),但验证层级事实性仍具挑战性。现有事实核查器主要针对通用领域原子性声明,缺乏测试其在DRRs中转移能力的基准。本文提出Evolving Benchmarking via Audit-then-Score(AtS)机制,使基准标签和理由可修订:当核查器与当前基准冲突时,需提交证据;审计员裁决争议;接受的修订更新基准后再对模型评分。在四轮AtS中,专家微金集准确率提升至90.9%,显示专家作为审计员比单次标注更可靠。DeepFact-Bench是可审计的DRR事实性基准,DeepFact-Eval是文档级验证智能体(含轻量变种),在DeepFact-Bench和外部事实性数据集上均表现优异。
英文摘要:
Search-augmented LLM agents can produce deep research reports (DRRs), but verifying claim-level factuality remains challenging. Existing fact-checkers are primarily designed for general-domain, factoid-style atomic claims, and there is no benchmark to test whether such verifiers transfer to DRRs. Yet building such a benchmark is itself difficult. We first show that static expert-labeled benchmarks are brittle in this setting: in a controlled study with PhD-level specialists, unassisted experts achieve only 60.8% accuracy on a hidden micro-gold set of verifiable claims. We propose Evolving Benchmarking via Audit-then-Score (AtS), where benchmark labels and rationales are explicitly revisable: when a verifier disagrees with the current benchmark, it must submit evidence; an auditor adjudicates the dispute; and accepted revisions update the benchmark before models are scored. Across four AtS rounds, expert micro-gold accuracy rises to 90.9%, indicating experts are substantially more reliable as auditors than as one-shot labelers. We instantiate AtS as DeepFact-Bench, a versioned DRR factuality benchmark with auditable rationales, and DeepFact-Eval, a document-level verification agent (with a grouped lite variant) that outperforms existing verifiers on DeepFact-Bench and transfers well to external factuality datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。