迈向事实性声明的高效提取与评估
Towards Effective Extraction and Evaluation of Factual Claims
AI总结：
本文针对LLM事实核查中声明提取缺乏标准化评估的问题，提出评估框架及自动化方法，并设计Claimify方法，在覆盖率和去语境化等指标上优于现有方法。
摘要：
对大型语言模型（LLMs）生成的长篇内容进行事实核查的一种常见策略是提取可独立验证的简单声明。由于不准确或不完整的声明会损害事实核查结果，确保声明质量至关重要。然而，缺乏标准化的评估框架阻碍了对声明提取方法的评估与比较。为填补这一空白，我们提出了一个在事实核查背景下评估声明提取的框架，以及应用该框架的自动化、可扩展且可复现的方法，包括衡量覆盖率和去语境化程度的新方法。我们还引入了Claimify，一种基于LLM的声明提取方法，并证明在我们的评估框架下，它优于现有方法。Claimify的一个关键特性是它能够处理歧义，并且仅在对源文本的正确解释具有高置信度时才提取声明。
英文摘要：
A common strategy for fact-checking long-form content generated by Large Language Models (LLMs) is extracting simple claims that can be verified independently. Since inaccurate or incomplete claims compromise fact-checking results, ensuring claim quality is critical. However, the lack of a standardized evaluation framework impedes assessment and comparison of claim extraction methods. To address this gap, we propose a framework for evaluating claim extraction in the context of fact-checking along with automated, scalable, and replicable methods for applying this framework, including novel approaches for measuring coverage and decontextualization. We also introduce Claimify, an LLM-based claim extraction method, and demonstrate that it outperforms existing methods under our evaluation framework. A key feature of Claimify is its ability to handle ambiguity and extract claims only when there is high confidence in the correct interpretation of the source text.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
