用反事实对抗幻觉：基于扩散引导的扰动用于LVLM幻觉抑制
Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression
AI总结：
CIPHER通过反事实图像扰动减少LVLM中的视觉幻觉，提升模型忠实性。
摘要：
尽管大型视觉-语言模型（LVLMs）在多模态任务中表现出色，但它们经常生成幻觉——与视觉输入不一致的不忠实输出。为了解决这个问题，我们引入了CIPHER（用于幻觉提取和消除的反事实图像扰动），这是一种无需训练的方法，通过轻量级的特征级校正来抑制视觉引起的幻觉。不同于以往无需训练的方法主要关注文本引起的幻觉，CIPHER明确针对由视觉模态引起的幻觉。CIPHER分为两个阶段。在离线阶段，我们构建OHC-25K（对象幻觉反事实，25,000个样本），一个由扩散编辑的图像组成的反事实数据集，这些图像故意与原始真实描述相矛盾。我们将这些编辑过的图像与未改变的真实描述配对，并通过LVLM处理以提取与幻觉相关的表示。将这些表示与真实（图像，描述）对的表示进行对比，揭示了跨越低秩子空间的结构化、系统性偏移，该子空间表征了视觉引起的幻觉。在推理阶段，CIPHER通过将中间隐藏状态投影到该子空间之外来抑制幻觉。在多个基准测试中，实验表明CIPHER显著降低了幻觉率，同时保持了任务性能，证明了反事实视觉扰动在提高LVLM忠实性方面的有效性。代码和附加材料可在https://hamidreza-dastmalchi.github.io/cipher-cvpr2026/上获得。
英文摘要：
While large vision-language models (LVLMs) achieve strong performance on multimodal tasks, they frequently generate hallucinations -- unfaithful outputs misaligned with the visual input. To address this issue, we introduce CIPHER (Counterfactual Image Perturbations for Hallucination Extraction and Removal), a training-free method that suppresses vision-induced hallucinations via lightweight feature-level correction. Unlike prior training-free approaches that primarily focus on text-induced hallucinations, CIPHER explicitly targets hallucinations arising from the visual modality. CIPHER operates in two phases. In the offline phase, we construct OHC-25K (Object-Hallucinated Counterfactuals, 25,000 samples), a counterfactual dataset consisting of diffusion-edited images that intentionally contradict the original ground-truth captions. We pair these edited images with the unchanged ground-truth captions and process them through an LVLM to extract hallucination-related representations. Contrasting these representations with those from authentic (image, caption) pairs reveals structured, systematic shifts spanning a low-rank subspace characterizing vision-induced hallucination. In the inference phase, CIPHER suppresses hallucinations by projecting intermediate hidden states away from this subspace. Experiments across multiple benchmarks show that CIPHER significantly reduces hallucination rates while preserving task performance, demonstrating the effectiveness of counterfactual visual perturbations for improving LVLM faithfulness. Code and additional materials are available at https://hamidreza-dastmalchi.github.io/cipher-cvpr2026/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
