通过解读注意力因果关系缓解多模态大语言模型中模态先验引发的幻觉
Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
AI总结：
本文提出 CausalMM 因果推断框架，通过视觉与语言注意力层面的后门调整和反事实推理消除模态先验混杂，显著缓解 MLLM 多模态幻觉。
摘要：
多模态大语言模型（MLLMs）已成为工业界和学术界的核心焦点，但往往受到视觉与语言先验所引入偏差的困扰，这些偏差可能导致多模态幻觉。这些偏差源于视觉编码器和大语言模型（LLM）主干，影响负责对齐多模态输入的注意力机制。现有基于解码的缓解方法关注统计相关性，却忽视注意力机制与模型输出之间的因果关系，从而限制了其处理这些偏差的有效性。为解决该问题，我们提出名为 CausalMM 的因果推断框架，将结构因果建模应用于 MLLMs，并把模态先验视为注意力机制与输出之间的混杂因素。具体而言，通过在视觉和语言两个注意力层面采用后门调整与反事实推理，我们的方法缓解了模态先验的负面影响，并增强 MLLM 输入与输出的对齐；与传统方法相比，其在 6 项 VLind-Bench 指标上最高取得 65.3% 的分数提升，并在 MME Benchmark 上提升 164 分。大量实验验证了该方法的有效性，同时它是一种即插即用的解决方案。我们的代码可在 https://github.com/The-Martyr/CausalMM 获取。
英文摘要：
Multimodal Large Language Models (MLLMs) have emerged as a central focus in both industry and academia, but often suffer from biases introduced by visual and language priors, which can lead to multimodal hallucination. These biases arise from the visual encoder and the Large Language Model (LLM) backbone, affecting the attention mechanism responsible for aligning multimodal inputs. Existing decoding-based mitigation methods focus on statistical correlations and overlook the causal relationships between attention mechanisms and model output, limiting their effectiveness in addressing these biases. To tackle this issue, we propose a causal inference framework termed CausalMM that applies structural causal modeling to MLLMs, treating modality priors as a confounder between attention mechanisms and output. Specifically, by employing backdoor adjustment and counterfactual reasoning at both the visual and language attention levels, our method mitigates the negative effects of modality priors and enhances the alignment of MLLM's inputs and outputs, with a maximum score improvement of 65.3% on 6 VLind-Bench indicators and 164 points on MME Benchmark compared to conventional methods. Extensive experiments validate the effectiveness of our approach while being a plug-and-play solution. Our code is available at: https://github.com/The-Martyr/CausalMM
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
