See Different, Think Better: 视觉变化缓解大视觉语言模型中的幻觉现象
See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs
AI总结：
针对大视觉语言模型易出现幻觉、现有文本中心方法在细粒度视觉场景下效果受限的问题，提出以视觉为中心的ViHallu框架，通过生成可控视觉变化图像结合定制视觉指令微调增强视觉-语义对齐，显著减少幻觉并提升细粒度理解能力，还发布了专用数据集。
摘要：
大视觉语言模型（LVLMs）在视觉理解和多模态推理方面已展现出卓越能力。然而，LVLMs频繁出现幻觉现象，表现为生成的文本回复与提供的视觉内容不一致。现有的幻觉缓解方法主要以文本为中心，视觉-语义对齐的挑战极大限制了其有效性，尤其是在面对细粒度视觉理解场景时。为此，本文提出ViHallu，一种以视觉为中心的幻觉缓解框架，通过视觉变化图像生成和视觉指令构建来增强视觉-语义对齐。ViHallu引入具有可控视觉改动的视觉变化图像，同时保留整体图像结构。这些图像与精心构建的视觉指令相结合，使LVLMs能够通过微调更好地理解细粒度视觉内容，让模型更精准地捕捉视觉内容与文本之间的对应关系，从而增强视觉-语义对齐。在多个基准上的大量实验表明，ViHallu有效增强了模型的细粒度视觉理解能力，同时显著降低了幻觉倾向。此外，我们发布了ViHallu-Instruction，这是一个专门为幻觉缓解和视觉-语义对齐设计的视觉指令数据集。代码可在https://github.com/oliviadzy/ViHallu获取。
英文摘要：
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in visual understanding and multimodal reasoning. However, LVLMs frequently exhibit hallucination phenomena, manifesting as the generated textual responses that demonstrate inconsistencies with the provided visual content. Existing hallucination mitigation methods are predominantly text-centric, the challenges of visual-semantic alignment significantly limit their effectiveness, especially when confronted with fine-grained visual understanding scenarios. To this end, this paper presents ViHallu, a Vision-Centric Hallucination mitigation framework that enhances visual-semantic alignment through Visual Variation Image Generation and Visual Instruction Construction. ViHallu introduces visual variation images with controllable visual alterations while maintaining the overall image structure. These images, combined with carefully constructed visual instructions, enable LVLMs to better understand fine-grained visual content through fine-tuning, allowing models to more precisely capture the correspondence between visual content and text, thereby enhancing visual-semantic alignment. Extensive experiments on multiple benchmarks show that ViHallu effectively enhances models' fine-grained visual understanding while significantly reducing hallucination tendencies. Furthermore, we release ViHallu-Instruction, a visual instruction dataset specifically designed for hallucination mitigation and visual-semantic alignment. Code is available at https://github.com/oliviadzy/ViHallu.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
