arXiv 2602.187112026-02-21cs.CV

HIME: 通过幻觉不敏感模型编辑缓解LVLMs中的物体幻觉

HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing

School of Information and Communication Technology, Griffith University, Australia（信息与通信技术学院，格里菲斯大学）；Data61, CSIRO, Australia（Data61，澳大利亚联邦科学与工业研究组织）；School of Engineering, Macquarie University, Sydney, Australia（工程学院，麦觉大学）；School of Information Systems, Queensland University of Technology, Australia（信息系统学院，昆士兰技术大学）

Ahmed Akl, Abdelwahed Khamis, Ali Cheraghian, Zhe Wang, Sara Khalifa, Kewen Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

HIME通过分层加权编辑方法有效抑制LVLMs中的物体幻觉，减少61.8%的幻觉问题，无需额外参数或计算开销。

摘要：

大型视觉-语言模型（LVLMs）已展示了令人印象深刻的多模态理解能力，但它们仍然容易产生物体幻觉，即模型描述不存在的物体或错误的事实信息，这对可靠的实际部署提出了严重担忧。虽然微调是常用的缓解策略，但其高计算成本和实际困难促使了训练自由替代方案的需求，其中模型编辑最近出现为有希望的方向。然而，随意的编辑可能会破坏预训练LVLMs中丰富的隐式知识，导致一个根本问题：在每一层需要多少干预才能抑制幻觉同时保留预训练知识？为了解决这个问题，我们系统地分析了基于三种广泛使用的大型语言模型骨干网络（Qwen、LLaMA和Vicuna）构建的LVLM解码器，揭示了在物体幻觉易感性方面的清晰分层差异。基于这些见解，我们引入了幻觉不敏感分数（HIS），一个原则性的度量标准，量化每一层对幻觉的敏感性，并为有针对性的干预提供指导。利用HIS，我们提出了幻觉不敏感模型编辑（HIME），一种简单但有效的分层加权编辑方法，通过选择性地修改潜在特征来抑制幻觉，同时保留预训练知识。广泛的实验表明，HIME在开放生成基准测试中平均减少了61.8%的幻觉，包括CHAIR、MME和GPT-4V辅助评估，而无需引入额外参数、推理时间延迟或计算开销。

英文摘要：

Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal understanding capabilities, yet they remain prone to object hallucination, where models describe non-existent objects or attribute incorrect factual information, raising serious concerns for reliable real-world deployment. While fine-tuning is a commonly adopted mitigation strategy, its high computational cost and practical difficulty motivate the need for training-free alternatives, among which model editing has recently emerged as a promising direction. However, indiscriminate editing risks disrupting the rich implicit knowledge encoded in pre-trained LVLMs, leading to a fundamental question: how much intervention is necessary at each layer to suppress hallucinations while preserving pre-trained knowledge? To address this question, we present a systematic analysis of LVLM decoders built on three widely used large language model backbones-Qwen, LLaMA, and Vicuna-revealing clear layer-wise differences in susceptibility to object hallucination. Building on these insights, we introduce the Hallucination Insensitivity Score (HIS), a principled metric that quantifies each layer's sensitivity to hallucination and provides guidance for targeted intervention. Leveraging HIS, we propose Hallucination Insensitivity Model Editing (HIME), a simple yet effective layer-adaptive weight editing approach that selectively modifies latent features to suppress hallucinations while preserving pre-trained knowledge. Extensive experiments demonstrate that HIME reduces hallucinations by an average of 61.8% across open-ended generation benchmarks, including CHAIR, MME, and GPT-4V-aided evaluation, without introducing additional parameters, inference-time latency, or computational overhead.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。