KG-FPQ：基于知识图谱的错误前提问题评估大型语言模型的事实性幻觉
KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questions
AI总结：
本文提出基于知识图谱的自动化流水线构建错误前提问题，并建立包含约178k问题的KG-FPQ基准，以评估大型语言模型的事实性幻觉并提供深刻见解。
摘要：
近期研究表明，大型语言模型（LLMs）容易受到错误前提问题（FPQs）的误导，导致事实知识出现错误，这被称为事实性幻觉。评估此脆弱性的现有基准主要依赖人工构建，导致规模有限且缺乏可扩展性。在本研究中，我们引入了一种基于知识图谱（KGs）的自动化、可扩展的FPQs构建流水线。第一步是修改从KGs中提取的真实三元组以创建错误前提。随后，利用GPTs的最先进能力，我们生成了语义丰富的FPQs。基于所提出的方法，我们提出了一个全面的基准——基于知识图谱的错误前提问题（KG-FPQ），它包含约178k个FPQs，涵盖三个知识领域、六个混淆级别和两种任务格式。使用KG-FPQ，我们对几个具有代表性的LLMs进行了广泛评估，并提供了有价值的见解。KG-FPQ数据集和代码可在 https://github.com/yanxuzhu/KG-FPQ 获取。
英文摘要：
Recent studies have demonstrated that large language models (LLMs) are susceptible to being misled by false premise questions (FPQs), leading to errors in factual knowledge, know as factuality hallucination. Existing benchmarks that assess this vulnerability primarily rely on manual construction, resulting in limited scale and lack of scalability. In this work, we introduce an automated, scalable pipeline to create FPQs based on knowledge graphs (KGs). The first step is modifying true triplets extracted from KGs to create false premises. Subsequently, utilizing the state-of-the-art capabilities of GPTs, we generate semantically rich FPQs. Based on the proposed method, we present a comprehensive benchmark, the Knowledge Graph-based False Premise Questions (KG-FPQ), which contains approximately 178k FPQs across three knowledge domains, at six levels of confusability, and in two task formats. Using KG-FPQ, we conduct extensive evaluations on several representative LLMs and provide valuable insights. The KG-FPQ dataset and code are available at~https://github.com/yanxuzhu/KG-FPQ.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
