在不降低性能的情况下缓解大型视觉-语言模型的幻觉
Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation
AI总结：
本文提出MPD框架，通过语义感知组件解耦和可解释参数更新，有效减少幻觉并保持生成能力，实验显示在LLaVA-Bench和MME上减少23.4%幻觉同时保持97.4%的生成能力。
摘要：
大型视觉-语言模型（LVLMs）具有强大的生成能力，但经常产生幻觉，影响输出可靠性。微调标注数据（无幻觉）是最直接的解决方案，但其高计算成本促使近期基于表示的方法，这些方法专注于减轻隐藏表示中的幻觉组件。尽管高效，我们发现这些方法由于无法完全提取幻觉组件和非选择性参数更新而降低了通用生成能力。为解决这些限制，我们提出MPD，一种双阶段框架，用于在不降低性能的情况下缓解幻觉。具体而言，我们的MPD依赖于两个关键因素：（1）语义感知组件解耦以提取纯幻觉组件，（2）可解释的参数更新，选择性地修改与幻觉最相关的参数。大量实验表明，MPD在LLaVA-Bench和MME上实现了最先进的性能，减少幻觉23.4%的同时保持97.4%的通用生成能力，且无额外计算成本。
英文摘要：
Large Vision-Language Models (LVLMs) exhibit powerful generative capabilities but frequently produce hallucinations that compromise output reliability. Fine-tuning on annotated data devoid of hallucinations offers the most direct solution, while its high computational cost motivates recent representation-based methods, which focus on mitigating hallucinatory components within hidden representations. Though efficient, we empirically observe that these methods degrade general generation capacity due to incomplete extraction of hallucination components and non-selective parameter updates. To address these limitations, we propose MPD, a dual-stage framework for mitigating hallucinations without performance degradation. Specifically, our MPD relies on two essential factors: (1) semantic-aware component disentanglement to extract pure hallucination components, and (2) interpretable parameter updates that selectively modify parameters most relevant to hallucination. Extensive experiments demonstrate that MPD achieves state-of-the-art performance, reducing hallucinations by 23.4\% while maintaining 97.4\% of general generative capability as evaluated on LLaVA-Bench and MME, with no additional computational cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
