KGHaluBench: 一种基于知识图谱的幻觉基准测试，用于评估大语言模型知识的广度和深度
KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge
AI总结：
KGHaluBench通过基于知识图谱的基准测试，评估大语言模型在知识广度和深度上的表现，提供更全面的真相评估方法。
摘要：
大型语言模型（LLMs）具有生成说服力和可理解语言的出色能力。然而，连贯性并不等于真实性，因为响应中往往包含微妙的幻觉。现有基准测试受限于静态和狭窄的问题，导致覆盖范围有限和误导性评估。我们提出了KGHaluBench，一种基于知识图谱的幻觉基准测试，评估LLM在知识广度和深度上的表现，提供更公平和全面的LLM真实性洞察。我们的框架利用KG动态构建具有挑战性和多方面的难题，其难度随后通过统计方法估算以解决流行偏差。我们的自动化验证流程检测回避并验证LLM的响应在概念和正确性层面以识别不同类型的幻觉。我们评估了25种前沿模型，使用新的准确性和幻觉度量标准。结果提供了更可解释的洞察，揭示了不同模型大小中导致幻觉的知识因素。KGHaluBench已公开可供未来幻觉缓解研究使用。
英文摘要：
Large Language Models (LLMs) possess a remarkable capacity to generate persuasive and intelligible language. However, coherence does not equate to truthfulness, as the responses often contain subtle hallucinations. Existing benchmarks are limited by static and narrow questions, leading to limited coverage and misleading evaluations. We present KGHaluBench, a Knowledge Graph-based hallucination benchmark that assesses LLMs across the breadth and depth of their knowledge, providing a fairer and more comprehensive insight into LLM truthfulness. Our framework utilises the KG to dynamically construct challenging, multifaceted questions, whose difficulty is then statistically estimated to address popularity bias. Our automated verification pipeline detects abstentions and verifies the LLM's response at both conceptual and correctness levels to identify different types of hallucinations. We evaluate 25 frontier models, using novel accuracy and hallucination metrics. The results provide a more interpretable insight into the knowledge factors that cause hallucinations across different model sizes. KGHaluBench is publicly available to support future developments in hallucination mitigation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
