基于谱表示滤波抑制视觉语言模型幻觉
Suppressing VLM Hallucinations with Spectral Representation Filtering
AI总结：
针对视觉语言模型易产生幻觉的问题，提出无需训练的轻量后置方法谱表示滤波SRF，零推理开销且无需改架构，在三类VLM的多个基准上有效降幻觉，实现最优保真度且不损描述质量。
摘要：
视觉语言模型（VLMs）因过度依赖语言先验且跨模态对齐不精确，常会生成幻觉内容，即描述图像中不存在的物体、属性或关系。本文提出谱表示滤波（Spectral Representation Filtering, SRF），这是一种轻量级、无需训练的方法，通过分析并校正模型表示的协方差结构来抑制此类幻觉。SRF对真实描述与幻觉描述对应特征的差值协方差做特征分解，识别出低秩幻觉模态，揭示特征空间中的结构化偏差；随后通过软谱滤波器在深层vLLM层的前馈投影权重中衰减这些模态，在保留语义保真度的同时均衡特征方差。与基于解码或重训练的方法不同，SRF完全以后置方式运行，零推理开销且无需修改模型架构。在LLaVA-1.5、MiniGPT-4、mPLUG-Owl2三类VLM上，SRF在MSCOCO、POPE-VQA及其他视觉任务基准上持续降低幻觉率，在不损害描述质量的前提下达到了当前最优的保真度水平。
英文摘要：
Vision-language models (VLMs) frequently produce hallucinations in the form of descriptions of objects, attributes, or relations that do not exist in the image due to over-reliance on language priors and imprecise cross-modal grounding. We introduce Spectral Representation Filtering (SRF), a lightweight, training-free method to suppress such hallucinations by analyzing and correcting the covariance structure of the model's representations. SRF identifies low-rank hallucination modes through eigendecomposition of the covariance of the differences between features collected for truthful and hallucinatory captions, revealing structured biases in the feature space. A soft spectral filter then attenuates these modes in the feed-forward projection weights of deeper vLLM layers, equalizing feature variance while preserving semantic fidelity. Unlike decoding or retraining-based approaches, SRF operates entirely post-hoc, incurs zero inference overhead, and requires no architectural modifications. Across three families of VLMs (LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2), SRF consistently reduces hallucination rates on MSCOCO, POPE-VQA, and other visual tasks benchmarks, achieving state-of-the-art faithfulness without degrading caption quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
