更仔细地看！一种对抗性参数编辑框架用于减轻视觉语言模型中的幻觉
Look Closer! An Adversarial Parametric Editing Framework for Hallucination Mitigation in VLMs
AI总结：
本文提出了一种对抗性参数编辑框架，通过激活-定位-编辑-对抗范式减轻视觉语言模型中的幻觉问题。
摘要：
尽管视觉语言模型（VLMs）由于其有前途的实际应用而在人工智能社区中受到越来越多的关注，但它们仍然存在持续的幻觉问题，生成的输出与视觉输入不一致。最近的研究将这些幻觉归因于VLMs过度依赖语言先验和不足的视觉特征整合，并提出了启发式解码校准策略来缓解这些问题。然而，这些策略的非可训练性质本质上限制了它们的优化潜力。为此，我们提出了一种对抗性参数编辑框架，用于减轻VLMs中的幻觉，该框架遵循A（激活）-L（定位）-E（编辑）-A（对抗）范式。具体来说，我们首先构建了一个激活数据集，其中包括基于现实的响应（正样本，紧密锚定在视觉特征上）和幻觉响应（负样本，反映LLM先验偏差和内部知识碎片）。接下来，我们通过分析响应对的差异隐藏状态来识别关键的幻觉易发参数簇。然后，这些簇通过注入带有对抗性调优前缀的提示进行微调，这些前缀被优化以最大化视觉忽视，从而迫使模型优先考虑视觉证据而非固有参数偏差。在生成和判别VLM任务上的评估显示了ALEAHallu在缓解幻觉方面的显著有效性。我们的代码可在https://github.com/hujiayu1223/ALEAHallu上获取。
英文摘要：
While Vision-Language Models (VLMs) have garnered increasing attention in the AI community due to their promising practical applications, they exhibit persistent hallucination issues, generating outputs misaligned with visual inputs. Recent studies attribute these hallucinations to VLMs' over-reliance on linguistic priors and insufficient visual feature integration, proposing heuristic decoding calibration strategies to mitigate them. However, the non-trainable nature of these strategies inherently limits their optimization potential. To this end, we propose an adversarial parametric editing framework for Hallucination mitigation in VLMs, which follows an \textbf{A}ctivate-\textbf{L}ocate-\textbf{E}dit \textbf{A}dversarially paradigm. Specifically, we first construct an activation dataset that comprises grounded responses (positive samples attentively anchored in visual features) and hallucinatory responses (negative samples reflecting LLM prior bias and internal knowledge artifacts). Next, we identify critical hallucination-prone parameter clusters by analyzing differential hidden states of response pairs. Then, these clusters are fine-tuned using prompts injected with adversarial tuned prefixes that are optimized to maximize visual neglect, thereby forcing the model to prioritize visual evidence over inherent parametric biases. Evaluations on both generative and discriminative VLM tasks demonstrate the significant effectiveness of ALEAHallu in alleviating hallucinations. Our code is available at https://github.com/hujiayu1223/ALEAHallu.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
