arXiv 2510.183212025-10-21cs.CV

超越单一模型：通过自适应Token集成解码缓解多模态幻觉

Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding

Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学 Gallup 学院）；Department of Electrical and Computer Engineering, McGill University（麦吉尔大学电气与计算机工程系）；School of Statistics, Renmin University of China（中国人民大学统计学院）；Tencent Jarvis Lab（腾讯 Jarvis 实验室）；Medical Artificial Intelligence Lab, Westlake University（西湖大学医学人工智能实验室）

Jinlin Li, Yuran Wang, Yifei Yuan, Xiao Zhou, Yingying Zhang, Xixian Yong, Yefeng Zheng, Xian Wu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大视觉语言模型易出现物体幻觉的问题，提出无需训练的自适应Token集成解码框架，通过推理阶段多模型预测聚合与动态权重计算缓解幻觉，性能优于现有最优方法，为提升模型鲁棒性提供了新方向。

摘要：

大视觉语言模型（LVLMs）近期在图像描述、视觉问答等多模态任务中取得了亮眼成果，但仍易出现物体幻觉问题，即生成不存在或识别错误物体的描述。已有研究通过辅助训练目标或外部模块部分缓解了该问题，但在可扩展性、适配性与模型独立性方面仍存在挑战。针对这些局限，我们提出自适应Token集成解码（ATED），这是一个无需训练的Token级集成框架，通过在推理阶段聚合多个LVLM的预测结果来缓解幻觉。ATED会为每个模型动态计算基于不确定性的权重，反映各模型在每个解码步骤的可靠性，还整合了多样的解码路径以提升语境 grounding 能力与语义一致性。在标准幻觉检测基准上的实验表明，ATED显著优于现有最优方法，在降低幻觉的同时未损害生成流畅度与相关性。我们的研究凸显了自适应集成的优势，也为提升高风险应用中LVLM的鲁棒性指明了有前景的方向。代码可在https://github.com/jinlin2021/ATED获取。

英文摘要：

Large Vision-Language Models (LVLMs) have recently achieved impressive results in multimodal tasks such as image captioning and visual question answering. However, they remain prone to object hallucination -- generating descriptions of nonexistent or misidentified objects. Prior work has partially mitigated this via auxiliary training objectives or external modules, but challenges remain in terms of scalability, adaptability, and model independence. To address these limitations, we propose Adaptive Token Ensemble Decoding (ATED), a training-free, token-level ensemble framework that mitigates hallucination by aggregating predictions from multiple LVLMs during inference. ATED dynamically computes uncertainty-based weights for each model, reflecting their reliability at each decoding step. It also integrates diverse decoding paths to improve contextual grounding and semantic consistency. Experiments on standard hallucination detection benchmarks demonstrate that ATED significantly outperforms state-of-the-art methods, reducing hallucination without compromising fluency or relevance. Our findings highlight the benefits of adaptive ensembling and point to a promising direction for improving LVLM robustness in high-stakes applications. The code is available at https://github.com/jinlin2021/ATED.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。