从文章到前提:构建PrimeFacts,一种提取事实核查证据的方法和资源
From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence
AI总结:
本文提出PrimeFacts方法,通过提取事实核查文章中的细粒度证据,提升自动化验证系统的性能,其在证据检索和声明验证中表现出显著优势。
摘要:
事实核查文章包含丰富的支持证据和推理,但这些证据由于无结构的呈现方式,难以被自动验证系统利用。我们引入PrimeFacts,一种从完整事实核查文章中提取细粒度证据的方法和资源。我们编译了13,106篇PolitiFact文章,包含声明、裁决及所有引用来源,并识别出49,718个文章内超链接作为自然锚点,以定位关键证据。我们的框架利用大型语言模型(LLMs)将这些锚点句子重写为独立于上下文的前提,并研究了额外隐含证据的提取。在跨文章证据检索和声明验证的评估中,提取的前提显著提升了性能。去上下文化的证据具有更高的可检索性,在Mean Reciprocal Rank上达到高达30%的相对增益,且用于裁决预测的宏F1值比基线提高了10-20个点。这些增益在不同裁决粒度(2类 vs. 5类)和模型架构下保持一致。定性分析表明,去上下文化的前提仍忠实于原始来源。我们的工作强调了重用事实核查者证据用于自动化的重要性,并提供了一个大规模的结构化证据资源,来源于真实世界中的事实核查。
英文摘要:
Fact-checking articles encode rich supporting evidence and reasoning, yet this evidence remains largely inaccessible to automated verification systems due to unstructured presentation. We introduce PrimeFacts, a methodology and resource for extracting fine-grained evidence from full fact-checking articles. We compile 13,106 PolitiFact articles with claims, verdicts, and all referenced sources, and we identify 49,718 in-article hyperlinks as natural anchors to pinpoint key evidence. Our framework leverages large language models (LLMs) to rewrite these anchor sentences into stand-alone, context-independent premises and investigates the extraction of additional implicit evidence. In evaluations on cross-article evidence retrieval and claim verification, the extracted premises substantially improve performance. Decontextualized evidence yields higher retrievability, achieving up to a 30 percent relative gain in Mean Reciprocal Rank over verbatim sentences, and using the evidence for verdict prediction raises Macro-F1 by 10-20 points over the baseline. These gains are consistent across different verdict granularities (2-class vs. 5-class) and model architectures. A qualitative analysis indicates that the decontextualized premises remain faithful to the original sources. Our work highlights the promise of reusing fact-checkers' evidence for automation and provides a large-scale resource of structured evidence from real-world fact-checks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。