arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.14976cs.CVcs.LG
收藏

基于度量驱动强化学习的LVLM图像字幕生成中的对象幻觉减少

Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning

School of Electrical Engineering, Korea Advanced Institute of Science and Technology(韩国科学技术院电气工程学院)

Thanh Nguyen, Hee Suk Yoon, Tung Minh Luu, Eunseop Yoon, Chang D. Yoo

AI总结:

本文提出一种由NLP度量驱动的强化学习框架,通过引入OCS、KL正则化及轻量级PPO变体,在冻结LVLM主干上微调适配器,可减少LVLM图像字幕生成中的对象幻觉,在COCO、Visual Genome数据集上分别实现41%、29%的幻觉降低,且能提升字幕质量。

摘要:

大型视觉语言模型(LVLM)在视觉与语言联合理解领域表现出色,尤其在图像字幕生成任务中,但仍易出现对象幻觉问题,即描述给定图像中不存在的实体。此前的解决方案包括数据集增强的监督训练和基于人类反馈的强化学习,虽有成效但高度依赖昂贵的人力投入,可扩展性受限。为解决该问题,本文提出一种完全由自然语言处理(NLP)度量驱动的强化学习框架,用于减少基于LVLM的图像字幕生成中的对象幻觉。仅基于NLP度量优化强化学习虽可提升可扩展性,但会引发奖励黑客、反馈稀疏和训练成本高昂等缺陷。本文框架通过引入对象一致性分数(OCS,同时捕捉对象的正确性与覆盖度)、结合KL正则化以实现更密集的令牌级指导,以及设计一种轻量级PPO变体(在冻结的LVLM主干之上微调适配器),规避了上述缺陷,实现了效率与稳定性。在InstructBLIP基准上开展的大量实验表明,该框架具有有效性与可泛化性:在COCO数据集上实现了高达41%的显著幻觉减少,在Visual Genome数据集上实现了29%的幻觉减少;此外,该框架可通过整合BERTScore、METEOR等互补评估指标提升字幕质量。这些结果揭示了多模态学习领域的一项重要洞见:LVLM可利用度量反馈有效自我校正幻觉,推动对大幅减少人力投入的可扩展强化框架的深入探索。

英文摘要:

Large Vision-Language Models (LVLMs) have excelled in joint visual and language understanding, particularly in image captioning. However, LVLMs remain prone to object hallucination, describing non-existent entities in the given image. Previous remedies, including dataset-enhanced supervised training and reinforcement learning from human feedback, show promise but rely heavily on costly human involvement, limiting scalability. To address this, we propose a reinforcement learning framework to mitigate object hallucination in LVLM-based image captioning, driven entirely by natural language processing (NLP) metrics. While optimizing reinforcement learning solely on NLP metrics offers scalability, it also introduces pitfalls: reward hacking, sparse feedback, and costly training. Our framework sidesteps these by introducing an Object Consistency Score (OCS) capturing both correctness and coverage of objects, incorporating KL-regularization for denser token-level guidance, and designing a lightweight PPO variant that fine-tunes adapters atop a frozen LVLM backbone, achieving efficiency and stability. Extensive experiments on the InstructBLIP baseline demonstrate the effectiveness and generalizability of our framework, achieving a substantial hallucination reduction of up to 41\% on the COCO dataset and 29\% on the Visual Genome. Furthermore, the framework can enhance caption quality by incorporating complementary evaluation metrics such as BERTScore and METEOR. These results reveal a broader insight in multimodal learning: LVLMs can effectively self-correct hallucinations using metric feedback, motivating deeper exploration into scalable reinforcement frameworks that significantly reduce human effort.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑