PerturboLLaVA：通过扰动视觉训练减少多模态幻觉
PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training
AI总结：
本文提出PerturboLLaVA方法，通过对抗性扰动文本训练降低模型对语言先验的依赖，并引入HalFscore细粒度指标，有效减少多模态大语言模型在密集图像描述中的幻觉，提升描述准确性与基准性能。
摘要：
本文旨在解决多模态大语言模型（MLLMs）中的幻觉挑战，特别是在密集图像描述任务中。为应对这一挑战，我们指出现有指标缺乏在概念层面上精细衡量描述质量的能力。为此，我们引入HalFscore，这是一种基于语言图构建的新型指标，旨在从细粒度层面评估密集描述的准确性和完整性。此外，我们确定幻觉的根本原因是模型对其语言先验的过度依赖。为解决这一问题，我们提出PerturboLLaVA，该方法通过在训练过程中引入对抗性扰动的文本来降低模型对语言先验的依赖。该方法增强了模型对视觉输入的关注，有效减少了幻觉，并生成准确、基于图像的描述，且不增加额外的计算开销。PerturboLLaVA显著提高了生成描述的保真度，在处理多模态幻觉方面优于现有方法，并在通用多模态基准测试中取得了更好的性能。
英文摘要：
This paper aims to address the challenge of hallucinations in Multimodal Large Language Models (MLLMs) particularly for dense image captioning tasks. To tackle the challenge, we identify the current lack of a metric that finely measures the caption quality in concept level. We hereby introduce HalFscore, a novel metric built upon the language graph and is designed to evaluate both the accuracy and completeness of dense captions at a granular level. Additionally, we identify the root cause of hallucination as the model's over-reliance on its language prior. To address this, we propose PerturboLLaVA, which reduces the model's reliance on the language prior by incorporating adversarially perturbed text during training. This method enhances the model's focus on visual inputs, effectively reducing hallucinations and producing accurate, image-grounded descriptions without incurring additional computational overhead. PerturboLLaVA significantly improves the fidelity of generated captions, outperforming existing approaches in handling multimodal hallucinations and achieving improved performance across general multimodal benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
