NoiseBoost：通过噪声扰动缓解多模态大语言模型的幻觉问题
NoiseBoost: Alleviating Hallucination with Noise Perturbation for Multimodal Large Language Models
AI总结：
本文提出NoiseBoost方法，通过引入噪声特征扰动作为正则化器，平衡多模态大语言模型对视觉和语言标记的注意力，有效缓解幻觉问题，并支持半监督学习以提升性能。
摘要：
多模态大语言模型（MLLMs）基于大语言模型为理解视觉信息提供了强大机制。然而，MLLMs因产生幻觉而臭名昭著，特别是在为图像生成冗长详细的描述时。我们的分析表明，幻觉源于大语言模型固有的总结机制，导致对语言标记的过度依赖，同时忽视了视觉信息。本文提出NoiseBoost，一种广泛适用且简单的方法，通过整合噪声特征扰动来缓解MLLMs的幻觉。噪声扰动作为正则化器，促进视觉和语言标记之间注意力权重的均衡分布。尽管简单，NoiseBoost在监督微调和强化学习等常见训练策略中持续提升MLLMs性能。此外，NoiseBoost开创性地为MLLMs启用半监督学习，释放无标签数据的潜力。综合实验表明，NoiseBoost在人工评估中将密集描述准确率提高8.1%，并通过挖掘无标签数据以50%的数据量取得可比结果。
英文摘要：
Multimodal large language models (MLLMs) contribute a powerful mechanism to understanding visual information building on large language models. However, MLLMs are notorious for suffering from hallucinations, especially when generating lengthy, detailed descriptions for images. Our analysis reveals that hallucinations stem from the inherent summarization mechanism of large language models, leading to excessive dependence on linguistic tokens while neglecting vision information. In this paper, we propose NoiseBoost, a broadly applicable and simple method for alleviating hallucinations for MLLMs through the integration of noise feature perturbations. Noise perturbation acts as a regularizer, facilitating a balanced distribution of attention weights among visual and linguistic tokens. Despite its simplicity, NoiseBoost consistently enhances the performance of MLLMs across common training strategies, including supervised fine-tuning and reinforcement learning. Further, NoiseBoost pioneerly enables semi-supervised learning for MLLMs, unleashing the power of unlabeled data. Comprehensive experiments demonstrate that NoiseBoost improves dense caption accuracy by 8.1% with human evaluation and achieves comparable results with 50% of the data by mining unlabeled data. Code and models are available at https://kaiwu5.github.io/noiseboost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
