ESREAL：利用语义重建缓解视觉-语言模型中的幻觉
ESREAL: Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models
AI总结：
本文提出无监督框架 ESREAL，通过基于描述的语义图像重建、区域对齐与 token 级幻觉评分，并结合 PPO 惩罚幻觉 token，在无需图文对的情况下显著降低多个视觉-语言模型的幻觉。
摘要：
视觉-语言模型中的幻觉对其可靠性构成了重大挑战，尤其是在生成长描述时。现有方法在准确识别和缓解这些幻觉方面存在不足。为解决该问题，我们提出 ESREAL，这是一种新颖的无监督学习框架，旨在通过准确定位和惩罚幻觉 token 来抑制幻觉的生成。首先，ESREAL 根据生成的描述创建一幅重建图像，并将其相应区域与原始图像的区域对齐。这种语义重建有助于识别生成描述中 token 级幻觉的存在及其类型。随后，ESREAL 根据幻觉类型评估对齐区域之间的语义相似性，从而计算 token 级幻觉分数。最后，ESREAL 采用近端策略优化算法，根据 token 级幻觉分数有选择地惩罚幻觉 token。我们的框架在 CHAIR 指标上使 LLaVA、InstructBLIP 和 mPLUG-Owl2 的幻觉分别显著降低 32.81%、27.08% 和 7.46%。这种改进完全通过源自图像本身的信号实现，无需任何图像-文本对。
英文摘要：
Hallucinations in vision-language models pose a significant challenge to their reliability, particularly in the generation of long captions. Current methods fall short of accurately identifying and mitigating these hallucinations. To address this issue, we introduce ESREAL, a novel unsupervised learning framework designed to suppress the generation of hallucinations through accurate localization and penalization of hallucinated tokens. Initially, ESREAL creates a reconstructed image based on the generated caption and aligns its corresponding regions with those of the original image. This semantic reconstruction aids in identifying both the presence and type of token-level hallucinations within the generated caption. Subsequently, ESREAL computes token-level hallucination scores by assessing the semantic similarity of aligned regions based on the type of hallucination. Finally, ESREAL employs a proximal policy optimization algorithm, where it selectively penalizes hallucinated tokens according to their token-level hallucination scores. Our framework notably reduces hallucinations in LLaVA, InstructBLIP, and mPLUG-Owl2 by 32.81%, 27.08%, and 7.46% on the CHAIR metric. This improvement is achieved solely through signals derived from the image itself, without the need for any image-text pairs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
