我们真的需要外部工具来缓解幻觉吗？SIRA：共享前缀内部重构归因
Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution
AI总结：
SIRA通过内部构建对比参考减少视觉语言模型的幻觉，无需外部工具或额外计算，保持描述覆盖并降低开销。
摘要：
大型视觉语言模型（LVLMs）在语言先验主导弱或模糊视觉证据时常出现幻觉。现有对比解码方法通过比较原始图像与外部扰动视觉输入的预测来缓解这一问题，但此类参考可能引入非流形伪影并需要昂贵的额外前向传递。我们提出了SIRA，一种无需训练的内部对比解码框架，通过利用多模态转换器的分阶段信息流，在同一LVLM内构建对比参考。SIRA不删除视觉信息，而是首先让图像和文本标记通过共享前缀交互，形成对齐的多模态状态，保留提示解释、解码历史、位置结构和早期视觉基础。然后在后续转换器层中分叉一个反事实分支，其中对图像标记位置的注意力被屏蔽。该分支保留共享的多模态上下文，但缺乏持续访问细粒度视觉证据，产生以语言先验主导的内部参考用于标记级对比。在解码过程中，SIRA抑制那些在没有后期视觉访问时仍强的标记，并倾向于预测其优势依赖于完整视觉路径的预测。在POPE、CHAIR和AMBER上使用Qwen2.5-VL和LLaVA-v1.5的实验表明，SIRA在减少幻觉的同时保持描述覆盖，并且比双通道对比解码具有更低的开销。SIRA无需训练、外部验证者或扰动输入，并适用于具有白盒推理访问的开放权重LVLMs。
英文摘要：
Large vision-language models (LVLMs) often hallucinate when language priors dominate weak or ambiguous visual evidence. Existing contrastive decoding methods mitigate this problem by comparing predictions from the original image with those from externally perturbed visual inputs, but such references can introduce off-manifold artifacts and require costly extra forward passes. We propose SIRA, a training-free internal contrastive decoding framework that constructs a counterfactual reference inside the same LVLM by exploiting the staged information flow of multimodal transformers. Instead of removing visual information from the input, SIRA first lets image and text tokens interact through a shared prefix, forming an aligned multimodal state that preserves prompt interpretation, decoding history, positional structure, and early visual grounding. It then forks a counterfactual branch in later transformer layers, where attention to image-token positions is masked. This branch retains the shared multimodal context but lacks continued access to fine-grained visual evidence, yielding a language-prior-dominated internal reference for token-level contrast. During decoding, SIRA suppresses tokens that remain strong without late visual access and favors predictions whose advantage depends on the full visual pathway. Experiments on POPE, CHAIR, and AMBER with Qwen2.5-VL and LLaVA-v1.5 show that SIRA consistently reduces hallucinations while preserving descriptive coverage and incurring lower overhead than two-pass contrastive decoding. SIRA requires no training, external verifier, or perturbed input, and applies to open-weight LVLMs with white-box inference access.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
