SAVE：基于稀疏自编码器的视觉信息增强用于缓解物体幻觉
SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucination
AI总结：
SAVE通过引导模型沿稀疏自编码器潜在特征减少物体幻觉，提升视觉理解能力。
摘要：
尽管多模态大语言模型（MLLMs）已经取得了显著进展，但它们仍然容易受到语言先验和视觉信息丢失导致的物体幻觉的影响。为了解决这个问题，我们提出了SAVE（稀疏自编码器驱动的视觉信息增强），这是一个通过引导模型沿着稀疏自编码器（SAE）潜在特征来减轻幻觉的框架。一个二进制物体存在性问题回答探测器识别出最能代表模型视觉信息处理的SAE特征，称为视觉理解特征。引导模型沿着这些识别出的特征可以加强基于事实的视觉理解，并有效减少幻觉。凭借其简单的设计，SAVE在标准基准上优于最先进的无训练方法，实现了在CHAIR_S上的10%p提升，并在POPE和MMHal-Bench上实现了持续的提升。在多个模型和层上的广泛评估证实了我们方法的鲁棒性和泛化能力。进一步分析表明，沿着视觉理解特征引导可以抑制不确定物体标记的生成，并增加对图像标记的关注，从而减轻幻觉。代码已发布在https://github.com/wiarae/SAVE。
英文摘要：
Although Multimodal Large Language Models (MLLMs) have advanced substantially, they remain vulnerable to object hallucination caused by language priors and visual information loss. To address this, we propose SAVE (Sparse Autoencoder-Driven Visual Information Enhancement), a framework that mitigates hallucination by steering the model along Sparse Autoencoder (SAE) latent features. A binary object-presence question-answering probe identifies the SAE features most indicative of the model's visual information processing, referred to as visual understanding features. Steering the model along these identified features reinforces grounded visual understanding and effectively reduces hallucination. With its simple design, SAVE outperforms state-of-the-art training-free methods on standard benchmarks, achieving a 10\%p improvement in CHAIR\_S and consistent gains on POPE and MMHal-Bench. Extensive evaluations across multiple models and layers confirm the robustness and generalizability of our approach. Further analysis reveals that steering along visual understanding features suppresses the generation of uncertain object tokens and increases attention to image tokens, mitigating hallucination. Code is released at https://github.com/wiarae/SAVE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
