SAGE：面向sink的 grounded 解码用于多模态幻觉缓解
SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation
AI总结：
SAGE通过动态调节自注意力机制，实时监控 grounding 可靠性，有效缓解多模态幻觉问题，实验显示在MSCOCO和AMBER基准上取得显著提升。
摘要：
大规模视觉-语言模型（VLMs）常出现幻觉，生成与视觉输入不一致的内容。现有方法通常通过后处理过滤、额外训练目标或外部验证解决，但无法在幻觉发生时干预解码过程。本文提出SAGE，一种Sink-Aware Grounded Decoding框架，通过动态调节自注意力机制缓解幻觉。幻觉与注意力sink tokens（标点或功能token）密切相关，这些token尽管语义内容有限，却积累不成比例的注意力。SAGE利用这些token作为锚点，实时监控grounding可靠性。在每个sink触发时，方法从生成序列中提取语义概念，利用自注意力图和梯度属性估计其视觉grounding，并测量其空间一致性。基于此信号，自注意力分布被自适应地锐化或拓宽，以增强grounded区域或抑制不可靠区域。在多样化的幻觉基准上进行的广泛实验表明，SAGE在不需模型再训练或架构修改的情况下，显著减少了幻觉的同时保持描述覆盖度。在MSCOCO和AMBER上，SAGE在不同VLM架构中实现了平均相对提升10.65%和7.19%。
英文摘要：
Large vision-language models (VLMs) frequently suffer from hallucinations, generating content that is inconsistent with visual inputs. Existing methods typically address this problem through post-hoc filtering, additional training objectives, or external verification, but they do not intervene during the decoding process when hallucinations arise. In this work, we introduce SAGE, a Sink-Aware Grounded Decoding framework that mitigates hallucinations by dynamically modulating self-attention during generation. Hallucinations are strongly correlated with attention sink tokens - punctuation or function tokens that accumulate disproportionate attention despite carrying limited semantic content. SAGE leverages these tokens as anchors to monitor grounding reliability in real time. At each sink trigger, the method extracts semantic concepts from the generated sequence, estimates their visual grounding using both self-attention maps and gradient-based attribution, and measures their spatial agreement. Based on this signal, self-attention distributions are adaptively sharpened or broadened to reinforce grounded regions or suppress unreliable ones. Extensive experiments across diverse hallucination benchmarks demonstrate that SAGE consistently outperforms existing decoding strategies, achieving substantial reductions in hallucination while preserving descriptive coverage, without requiring model retraining or architectural modifications. Our method achieves an average relative improvement of 10.65% on MSCOCO and 7.19% on AMBER across diverse VLM architectures, demonstrating consistent gains in hallucination mitigation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
