通过放大镜：用于无幻觉VLM解码的自适应感知放大
Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding
AI总结：
本文提出感知放大器PM，通过迭代隔离相关视觉标记并放大相应区域，提升VLM对视觉细节的 scrutiny，从而减少幻觉并增强语言生成与推理能力。
摘要：
现有视觉-语言模型（VLMs）常面临视觉幻觉问题，即生成的响应包含不基于视觉输入的不准确信息。为解决此问题而不进行模型微调，现有方法主要通过对比性减少语言偏差或在解码过程中放大视觉嵌入权重来缓解幻觉。然而，这些方法在捕捉细粒度视觉细节方面仍有限。本文提出感知放大器（PM），一种新颖的视觉解码方法，通过迭代隔离相关视觉标记并放大相应区域，促使模型在解码过程中专注于细粒度视觉细节。通过在每个解码步骤中放大关键区域同时保留结构和上下文信息，PM使VLM能够增强对视觉输入的审视，从而生成更准确和忠实的响应。大量实验结果表明，PM不仅实现了更优异的幻觉缓解，还增强了语言生成能力，同时保持了强大的推理能力。代码可在https://github.com/ShunqiM/PM中找到。
英文摘要：
Existing vision-language models (VLMs) often suffer from visual hallucination, where the generated responses contain inaccuracies that are not grounded in the visual input. Efforts to address this issue without model finetuning primarily mitigate hallucination by contrastively reducing language biases or amplifying the weights of visual embedding during decoding. However, these approaches remain limited in their ability to capture fine-grained visual details. In this work, we propose the Perception Magnifier (PM), a novel visual decoding method that iteratively isolates relevant visual tokens based on attention and magnifies the corresponding regions, spurring the model to concentrate on fine-grained visual details during decoding. By magnifying critical regions while preserving the structural and contextual information at each decoding step, PM allows the VLM to enhance its scrutiny of the visual input, hence producing more accurate and faithful responses. Extensive experimental results demonstrate that PM not only achieves superior hallucination mitigation but also enhances language generation while preserving strong reasoning capabilities. Code can be found at https://github.com/ShunqiM/PM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
