MultiHal：用于知识图谱接地的大语言模型幻觉评估的多语言数据集
MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations
AI总结：
针对现有幻觉评估基准缺乏知识图谱路径和多语言支持的问题，提出多语言多跳基准MultiHal，通过挖掘并精选2.59万条知识图谱路径，在KG-RAG中显著提升语义相似度、NLI蕴含和幻觉检测得分，验证了知识图谱集成的潜力。
摘要：
大语言模型（LLMs）在忠实性和事实性方面存在固有局限，通常被称为幻觉。目前已开发出多个基准，在英语中心的数据集背景下为事实性评估提供测试平台，同时依赖网页链接或文本段落等补充性信息上下文，但忽略了可用的结构化事实资源。为此，知识图谱（KGs）已被认为是缓解幻觉的有用辅助工具，因为它们提供了一种结构化的方式来表示关于实体及其关系的事实，且语言开销最小。我们弥补了现有幻觉评估基准中缺乏知识图谱路径和多语言事实语言建模的不足，提出了一个基于知识图谱的多语言、多跳基准，名为MultiHal，用于生成式文本评估。作为数据收集流程的一部分，我们从开放域知识图谱中挖掘了14万条知识图谱路径，从中修剪了噪声知识图谱路径，精选出2.59万条高质量子集。我们的基线评估显示，在多种语言和多种模型下，知识图谱增强检索生成（KG-RAG）相较于普通问答（vanilla QA）在语义相似度得分上绝对提升了约0.12至0.36分，在自然语言推理（NLI）蕴含上提升了0.16至0.36分，在幻觉检测上提升了0.29至0.42分，展示了知识图谱集成的潜力。我们期望MultiHal将促进未来在基于图的各种幻觉缓解和事实核查任务上的研究。
英文摘要：
Large Language Models (LLMs) have inherent limitations of faithfulness and factuality, commonly referred to as hallucinations. Several benchmarks have been developed that provide a test bed for factuality evaluation within the context of English-centric datasets, while relying on supplementary informative context like web links or text passages but ignoring the available structured factual resources. To this end, Knowledge Graphs (KGs) have been identified as a useful aid for hallucination mitigation, as they provide a structured way to represent the facts about entities and their relations with minimal linguistic overhead. We bridge the lack of KG paths and multilinguality for factual language modeling within the existing hallucination evaluation benchmarks and propose a KG-based multilingual, multihop benchmark called MultiHal framed for generative text evaluation. As part of our data collection pipeline, we mined 140k KG-paths from open-domain KGs, from which we pruned noisy KG-paths, curating a high-quality subset of 25.9k. Our baseline evaluation shows an absolute scale improvement by approximately 0.12 to 0.36 points for the semantic similarity score, 0.16 to 0.36 for NLI entailment and 0.29 to 0.42 for hallucination detection in KG-RAG over vanilla QA across multiple languages and multiple models, demonstrating the potential of KG integration. We anticipate MultiHal will foster future research towards several graph-based hallucination mitigation and fact-checking tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
