HELPD：通过视觉增强惩罚解码的分层反馈学习缓解大视觉语言模型的幻觉
HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
AI总结：
本文提出HELPD框架，通过在物体和句子语义层面进行分层反馈学习，并结合视觉增强惩罚解码，以少量训练即可缓解超过15%的多模态幻觉，同时提升LVLMs的文本生成质量。
摘要：
大视觉语言模型（LVLMs）在许多视觉语言任务上表现出色，但这些模型仍然遭受多模态幻觉的困扰，即生成违反图像内容的物体或内容。许多现有工作通过直接判断物体是否存在于图像中来检测幻觉，忽视了物体与语义之间的关联。为解决这一问题，我们提出了具有视觉增强惩罚解码的分层反馈学习框架（HELPD）。该框架在物体和句子语义两个层面引入幻觉反馈。值得注意的是，即使仅进行少量训练，该方法也能缓解超过15%的幻觉。同时，HELPD根据图像注意力窗口对输出logits进行惩罚，以避免过度受生成文本的影响。HELPD可以无缝集成到任何LVLMs中。我们的实验表明，所提出的框架在多个幻觉基准上取得了良好的结果。它有效缓解了不同LVLMs的幻觉，并同时提升了它们的文本生成质量。
英文摘要：
Large Vision-Language Models (LVLMs) have shown remarkable performance on many visual-language tasks. However, these models still suffer from multimodal hallucination, which means the generation of objects or content that violates the images. Many existing work detects hallucination by directly judging whether an object exists in an image, overlooking the association between the object and semantics. To address this issue, we propose Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding (HELPD). This framework incorporates hallucination feedback at both object and sentence semantic levels. Remarkably, even with a marginal degree of training, this approach can alleviate over 15% of hallucination. Simultaneously, HELPD penalizes the output logits according to the image attention window to avoid being overly affected by generated text. HELPD can be seamlessly integrated with any LVLMs. Our experiments demonstrate that the proposed framework yields favorable results across multiple hallucination benchmarks. It effectively mitigates hallucination for different LVLMs and concurrently improves their text generation quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
