缓解视觉-语言模型中的图像描述幻觉
Mitigating Image Captioning Hallucinations in Vision-Language Models
AI总结：
针对 VLM 图像描述幻觉，本文提出仅更新语言模型层归一化少量参数的强化学习测试时适应框架，并用 CLIP 评估模型提供双重奖励，在 LLaVA 与 InstructBLIP 上显著降低幻觉率。
摘要：
视觉-语言模型（VLM）中的幻觉会损害可靠性和现实适用性，其通常源于预训练数据与测试样本之间的分布偏移。现有解决方案（例如在额外数据上重新训练或微调）需要大量计算资源和劳动密集型的数据收集，而基于集成的方法通过引入辅助 VLM 产生额外成本。为应对这些挑战，我们提出一种新颖的测试时适应框架，利用强化学习在推理过程中缓解幻觉，无需重新训练，也不使用任何辅助 VLM。通过仅更新语言模型层归一化中的可学习参数（约占模型参数的 0.003%），我们的方法减小了测试样本与预训练样本之间的分布偏移。文中还提出一个基于 CLIP 的幻觉评估模型，为 VLM 提供双重奖励。实验结果表明，在 LLaVA 和 InstructBLIP 上幻觉率分别降低 15.4% 和 17.3%。我们的方法优于最先进基线，幻觉缓解效果提升 68.3%，证明了其有效性。
英文摘要：
Hallucinations in vision-language models (VLMs) hinder reliability and real-world applicability, usually stemming from distribution shifts between pretraining data and test samples. Existing solutions, such as retraining or fine-tuning on additional data, demand significant computational resources and labor-intensive data collection, while ensemble-based methods incur additional costs by introducing auxiliary VLMs. To address these challenges, we propose a novel test-time adaptation framework using reinforcement learning to mitigate hallucinations during inference without retraining or any auxiliary VLMs. By updating only the learnable parameters in the layer normalization of the language model (approximately 0.003% of the model parameters), our method reduces distribution shifts between test samples and pretraining samples. A CLIP-based hallucination evaluation model is proposed to provide dual rewards to VLMs. Experimental results demonstrate a 15.4% and 17.3% reduction in hallucination rates on LLaVA and InstructBLIP, respectively. Our approach outperforms state-of-the-art baselines with a 68.3% improvement in hallucination mitigation, demonstrating its effectiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
