arXiv 2605.253772026-05-25cs.CVcs.AI

对抗正交解缠用于LVLM幻觉缓解

Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation

Fudan University（复旦大学）；Tencent（腾讯）；Nanjing University（南京大学）；Southeast University（东南大学）；Great Bay University（大坝大学）；TeleAI, China Telecom（TeleAI，中国电信）

Ruoxi Cheng, Haoxuan Ma, Zhengfei Hai, Yiyan Huang, Ranjie Duan, Tianle Zhang, Xu Yang, Ziyi Ye, Xingjun Ma

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出对抗正交解缠（AOD）框架，通过最小最大目标学习幻觉相关方向，并利用双前向对比解码策略，在不需额外训练的情况下缓解大型视觉语言模型（LVLM）的幻觉问题。

摘要：

大型视觉语言模型（LVLM）推进了多模态理解，但其可靠性受到幻觉的限制，即生成内容与视觉事实冲突。现有缓解方法要么依赖昂贵的外部干预（如指令调优和检索），要么使用受限于有缺陷的注意力权重和纠缠的隐藏表示的内部机制。我们提出对抗正交解缠（AOD），一种用于缓解LVLM幻觉的潜在几何框架。AOD通过最小最大目标学习幻觉相关方向：分类器将幻觉信号集中到投影分量中，而对抗器通过梯度反转层将其从正交残差空间中移除。学习到的方向使得一种无需训练的双前向对比解码策略能够抑制幻觉同时保持通用能力。在三个LVLM上进行的四个幻觉和四个效用基准实验表明，AOD一致优于强基线。它在POPE上平均提高超过6%的准确率，将AMBER提升6%，并在MMMU等效用任务上保持强劲性能。进一步分析显示跨数据集的鲁棒迁移，表明AOD捕获了通用的幻觉相关偏差而非数据集特定伪影。我们的源代码和数据集可在https://github.com/Hunter-Wrynn/AOD获取。

英文摘要：

Large Vision-Language Models (LVLMs) have advanced multimodal understanding, yet their reliability is limited by hallucination, where generated content conflicts with visual facts. Existing mitigation methods either rely on costly external interventions, such as instruction tuning and retrieval, or use internal mechanisms that remain limited by flawed attention weights and entangled hidden representations. We propose Adversarial Orthogonal Disentanglement (AOD), a latent geometric framework for mitigating LVLM hallucinations. AOD learns a hallucination-related direction through a minimax objective: a classifier concentrates hallucination signals into the projected component, while an adversary removes them from the orthogonal residual space via a Gradient Reversal Layer. The learned direction enables a training-free dual-forward-pass contrastive decoding strategy that suppresses hallucinations while preserving general capabilities. Experiments on three LVLMs across four hallucination and four utility benchmarks show that AOD consistently outperforms strong baselines. It improves POPE accuracy by over 6\% on average, boosts AMBER by 6\%, and maintains strong performance on utility tasks such as MMMU. Further analysis shows robust transfer across datasets, suggesting that AOD captures general hallucination-related biases rather than dataset-specific artifacts. Our source code and datasets are available at https://github.com/Hunter-Wrynn/AOD.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。