通过推理时相关性传播缓解多模态大语言模型的幻觉
Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time
AI总结：
本文提出基于逐层相关性传播的无需训练框架LIME，在推理时优化键值表示以增强感知输入相关性，从而缓解多模态大语言模型的幻觉，并在视觉和音频基准上验证了效果。
摘要：
多模态大语言模型（MLLMs）在视觉和音频语言任务上表现出色，但可能生成与给定视觉或听觉输入相冲突的响应，这一问题被称为多模态幻觉。先前的工作表明，当模型更多地依赖文本线索和学习到的语言模式而非感知输入中的证据时，就会出现这种情况。为了更直接地解释这种不平衡，我们应用逐层相关性传播（LRP），将预测归因于单个输入令牌，并利用所得的相关性分数来分析和缓解幻觉。首先，我们考察这种不平衡是否导致多模态幻觉。我们发现，当模型较少依赖感知输入时，幻觉常常出现，而改变这种依赖会影响其预测。我们进一步利用LRP，提出一个无需训练的框架，该框架在解码过程中通过优化键值表示将相关性转向感知令牌，而不修改模型参数或需要训练数据。我们将此方法称为学习推理时模态增强（LIME）。尽管不使用空间或时间监督，LIME将相关性集中在查询相关区域。我们在视觉和音频领域的多个多模态基准上评估LIME，证明其在保持生成质量的同时，持续减少幻觉并增强接地性。
英文摘要：
Multimodal large language models (MLLMs) achieve strong performance on vision- and audio-language tasks, yet can generate responses that conflict with the given visual or auditory inputs, a problem known as multimodal hallucinations. Prior work suggests that this occurs when models rely more on textual cues and learned language patterns than on evidence from the perceptual input. To obtain a more direct account of this imbalance, we apply Layer-wise Relevance Propagation (LRP), which attributes predictions to individual input tokens, and use the resulting relevance scores to analyze and mitigate hallucinations. First, we examine whether this imbalance leads to multimodal hallucinations. We find that hallucinations often arise when the model relies less on perceptual inputs, and that changing this reliance affects its predictions. We further leverage LRP and propose a training-free framework that shifts relevance toward perceptual tokens by optimizing key-value representations during decoding, without modifying model parameters or requiring training data. We call this method Learning Inference-time Modality Enhancement (LIME). Despite using no spatial or temporal supervision, LIME concentrates relevance on query-relevant regions. We evaluate LIME across multiple multimodal benchmarks in both vision and audio domains, demonstrating consistent reductions in hallucinations and enhanced grounding while preserving generation quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
