arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.03141cs.CL
收藏

超越精确度:面向长文本LLM生成的事实性评估中的重要性感知召回

Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation

UMass Amherst(马萨诸塞大学阿默斯特分校);University of Maryland(马里兰大学)

Nazanin Jafari, James Allan, Mohit Iyyer

AI总结:

本文提出一种综合衡量精确度与召回的框架,通过外部知识源构建参考事实,并引入基于相关性和显著性的权重方案,揭示当前LLM在长文本生成中事实性不完整的问题。

摘要:

本文提出一种综合衡量精确度与召回的框架,通过外部知识源构建参考事实,并引入基于相关性和显著性的权重方案,揭示当前LLM在长文本生成中事实性不完整的问题。

英文摘要:

Evaluating the factuality of long-form output generated by large language models (LLMs) remains challenging, particularly when responses are open-ended and contain many fine-grained factual statements. Existing evaluation methods primarily focus on precision: they decompose a response into atomic claims and verify each claim against external knowledge sources such as Wikipedia. However, this overlooks an equally important dimension of factuality: recall, whether the generated response covers the relevant facts that should be included. We propose a comprehensive factuality evaluation framework that jointly measures precision and recall. Our method leverages external knowledge sources to construct reference facts and determine whether they are captured in generated text. We further introduce an importance-aware weighting scheme based on relevance and salience. Our analysis reveals that current LLMs perform substantially better on precision than on recall, suggesting that factual incompleteness remains a major limitation of long-form generation and that models are generally better at covering highly important facts than the full set of relevant facts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑