Evaluating the factuality of long-form output generated by large language models (LLMs) remains challenging, particularly when responses are open-ended and contain many fine-grained factual statements. Existing evaluation methods primarily focus on precision: they decompose a response into atomic claims and verify each claim against external knowledge sources such as Wikipedia. However, this overlooks an equally important dimension of factuality: recall, whether the generated response covers the relevant facts that should be included. We propose a comprehensive factuality evaluation framework that jointly measures precision and recall. Our method leverages external knowledge sources to construct reference facts and determine whether they are captured in generated text. We further introduce an importance-aware weighting scheme based on relevance and salience. Our analysis reveals that current LLMs perform substantially better on precision than on recall, suggesting that factual incompleteness remains a major limitation of long-form generation and that models are generally better at covering highly important facts than the full set of relevant facts.
MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100085, China(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China(中国科学院大学网络安全学院)
Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
Department of Computer Science and Engineering, United International University, Dhaka 1212, Bangladesh(乌姆特国际大学计算机科学与工程系);Department of Computer Science and Engineering, Daffodil International University, Dhaka-1341, Bangladesh(达芙尼国际大学计算机科学与工程系);Faculty of Science and Technology, Charles Darwin University, Casuarina, NT 0909, Australia(查尔斯·达尔文大学科学与技术学院)
KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);The University of Osaka(大阪大学)