暴露幻觉以抑制幻觉：基于生成锚点的视觉语言模型表示编辑
Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors
AI总结：
本文提出一种无需训练的自监督方法，通过文本到图像模型放大幻觉信号形成负锚点，并以原始图像为正锚点编辑解码器表示，从而在保持描述丰富度的同时显著抑制多模态大语言模型在物体、属性和关系层面的幻觉。
摘要：
多模态大语言模型（MLLMs）在各类视觉-语言任务中取得了显著成功，但它们仍然极易产生幻觉，生成流畅但与视觉证据不一致的内容。这类幻觉涵盖物体、属性和关系层面，即使在更大的模型中依然存在，而现有的缓解方法通常需要额外的微调、手工设计的先验知识，或以牺牲信息量和可扩展性为代价。为了解决这一局限，我们提出了一种无需训练、自监督的幻觉缓解方法。我们的方法引入了一种新颖的幻觉放大机制：通过文本到图像模型将描述文本投影到视觉空间，以揭示隐式的幻觉信号，并将其作为负锚点，而原始图像则作为正锚点。利用这对双重锚点，我们通过将表示拉向忠实语义并推离幻觉方向来编辑解码器隐藏状态。这种校正不需要人工先验知识，也不增加额外的训练成本，从而保证了有效性和高效性。在多个基准上的大量实验表明，我们的方法在物体、属性和关系层面显著减少了幻觉，同时基本保持了召回率和描述丰富度，例如在 CHAIR 上使用 LLaVA-v1.5-7B 实现了超过 5% 的幻觉减少。此外，在包括 LLaVA-NEXT-7B、Cambrian-8B 和 InstructBLIP-7B 在内的多种架构上的结果验证了强大的跨架构泛化能力。更重要的是，当应用于无幻觉的描述文本时，我们的方法几乎不引入副作用，凸显了其鲁棒性和实用的即插即用适用性。实现代码将公开提供。
英文摘要：
Multimodal large language models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet they remain highly susceptible to hallucinations, producing content that is fluent but inconsistent with visual evidence. Such hallucinations, spanning objects, attributes, and relations, persist even in larger models, while existing mitigation approaches often require additional finetuning, handcrafted priors, or trade-offs that compromise informativeness and scalability. To address this limitation, we propose a training-free, self-supervised method for hallucination mitigation. Our approach introduces a novel hallucination amplification mechanism: a caption is projected into the visual space via a text-to-image model to reveal implicit hallucination signals, serving as a negative anchor, while the original image provides a positive anchor. Leveraging these dual anchors, we edit decoder hidden states by pulling representations toward faithful semantics and pushing them away from hallucination directions. This correction requires no human priors or additional training costs, ensuring both effectiveness and efficiency. Extensive experiments across multiple benchmarks show that our method significantly reduces hallucinations at the object, attribute, and relation levels while largely preserving recall and caption richness, e.g., achieving a hallucination reduction by over 5% using LLaVA-v1.5-7B on CHAIR. Furthermore, results on diverse architectures, including LLaVA-NEXT-7B, Cambrian-8B, and InstructBLIP-7B, validate strong cross-architecture generalization. More importantly, when applied to hallucination-free captions, our method introduces almost no side effects, underscoring its robustness and practical plug-and-play applicability. The implementation will be publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
