MAD：用于减轻多模态大语言模型中跨模态幻觉的模态自适应解码
MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
AI总结：
MAD通过自适应加权对比解码分支，有效减少多模态大语言模型中的跨模态幻觉问题。
摘要：
多模态大语言模型（MLLMs）存在跨模态幻觉问题，即一种模态不当地影响另一种模态的生成，导致生成内容虚假。这暴露了模态交互控制更基本的缺陷。为了解决这一问题，我们提出了模态自适应解码（MAD），这是一种无需训练的方法，根据任务需求自适应地加权模态特定的解码分支。MAD利用模型自身评估模态相关性的能力，通过查询每个任务需要哪些模态来确定。提取的模态概率随后用于自适应加权对比解码分支，使模型能够专注于相关信息并抑制跨模态干扰。在CMM和AVHBench上的广泛实验表明，MAD在多个音频-视觉语言模型中显著减少了跨模态幻觉（对于VideoLLaMA2-AV，改进了7.8%和2.0%；对于Qwen2.5-Omni，改进了8.7%和4.7%）。我们的方法证明了通过自我评估实现显式模态意识对于稳健的多模态推理至关重要，为现有的对比解码方法提供了原则性的扩展。我们的代码可在https://github.com/top-yun/MAD获取。
英文摘要：
Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8\% and 2.0\% improvements for VideoLLaMA2-AV, 8.7\% and 4.7\% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at \href{https://github.com/top-yun/MAD}{https://github.com/top-yun/MAD}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
