arXiv 2407.004882024-06-29cs.CLcs.AI

PFME：一种用于大语言模型细粒度幻觉检测与编辑的模块化方法

PFME: A Modular Approach for Fine-grained Hallucination Detection and Editing of Large Language Models

Beihang University（北京航空航天大学）；The University of Edinburgh（爱丁堡大学）；University of Manchester（曼彻斯特大学）；Chongqing University（重庆大学）

Kunquan Deng, Zeyu Huang, Chen Li, Chenghua Lin, Min Gao, Wenge Rong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出渐进式细粒度模型编辑器PFME，通过实时事实检索与句子级幻觉检测编辑两个模块，在FavaBench和FActScore上显著提升LLM细粒度幻觉检测与修正性能。

摘要：

大语言模型（LLMs）在流畅性方面表现出色，但存在生成不准确内容（即“幻觉”）的风险。本文概述了一种用于对细粒度幻觉类型进行分类的标准化流程，并提出了一种创新框架——渐进式细粒度模型编辑器（PFME），专门用于检测和纠正LLMs中的细粒度幻觉。PFME由两个协作模块组成：实时事实检索模块和细粒度幻觉检测与编辑模块。前者识别文档中的关键实体，并从可信来源检索最新的事实证据。后者进一步将文档分割为句子级文本，并基于相关证据和先前已编辑的上下文，识别、定位并编辑每个句子的幻觉类型。在FavaBench和FActScore上的实验结果表明，PFME在细粒度幻觉检测任务中优于现有方法。特别是，当使用Llama3-8B-Instruct模型时，PFME在外部知识辅助下的细粒度幻觉检测性能相比ChatGPT提高了8.7个百分点（pp）。在编辑任务中，PFME进一步提升了FActScore-Alpaca13B和FActScore-ChatGPT数据集的FActScore，分别提高了16.2个百分点和4.6个百分点。

英文摘要：

Large Language Models (LLMs) excel in fluency but risk producing inaccurate content, called "hallucinations." This paper outlines a standardized process for categorizing fine-grained hallucination types and proposes an innovative framework--the Progressive Fine-grained Model Editor (PFME)--specifically designed to detect and correct fine-grained hallucinations in LLMs. PFME consists of two collaborative modules: the Real-time Fact Retrieval Module and the Fine-grained Hallucination Detection and Editing Module. The former identifies key entities in the document and retrieves the latest factual evidence from credible sources. The latter further segments the document into sentence-level text and, based on relevant evidence and previously edited context, identifies, locates, and edits each sentence's hallucination type. Experimental results on FavaBench and FActScore demonstrate that PFME outperforms existing methods in fine-grained hallucination detection tasks. Particularly, when using the Llama3-8B-Instruct model, PFME's performance in fine-grained hallucination detection with external knowledge assistance improves by 8.7 percentage points (pp) compared to ChatGPT. In editing tasks, PFME further enhances the FActScore of FActScore-Alpaca13B and FActScore-ChatGPT datasets, increasing by 16.2pp and 4.6pp, respectively.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。