LISA：一种用于缓解多模态大语言模型幻觉的逐层集成与抑制方法
LISA: A Layer-wise Integration and Suppression Approach for Hallucination Mitigation in Multimodal Large Language Models
AI总结：
针对多模态大语言模型易产生物体幻觉的问题，提出即插即用的逐层集成与抑制方法LISA，通过频谱调制与锚点路由融合logits缓解幻觉，在多个基准上取得显著效果且泛化性强。
摘要：
多模态大语言模型（MLLMs）在图像描述等视觉-语言任务上表现优异，但仍易出现物体幻觉问题，即会描述图像中并未出现的物体。为缓解该问题，我们提出了逐层集成与抑制方法LISA。LISA利用了MLLMs的逐层功能特性：浅层提供视觉定位能力，中层编码语义信息，深层则容易放大虚假信号。首先，逐层频谱调制通过抑制深层中过度放大的激活值，同时保留浅层的对齐线索，来稳定注意力机制。其次，选定层输出的token级logits通过基于锚点的路由进行融合，其中逐token锚点选择与软logit融合可在解码过程中实现自适应集成。LISA是完全即插即用的方法，可无缝集成到包括Qwen2.5-VL在内的现有MLLMs中。在多个基准数据集上的实验表明，LISA在CHAIR_I指标上最高可将幻觉降低53.6%，在POPE F1指标上最高可提升5.1%，展现出跨模型、跨任务的强泛化性。我们的代码可在https://github.com/zhlisa1010-eng/LISA获取。
英文摘要：
Multimodal Large Language Models (MLLMs) excel in vision-language tasks such as image captioning but remain prone to object hallucinations, where they describe objects that do not appear in the image. To mitigate this, we propose LISA, a Layer-wise Integration and Suppression Approach. LISA leverages the layer-wise functional roles in MLLMs: shallow layers provide visual grounding, middle layers encode semantics, and deep layers tend to amplify spurious signals. First, layer-wise spectral modulation stabilizes attention by suppressing over-amplified activations in deeper layers while preserving alignment cues in earlier layers. Second, token-level logits from selected layers are fused via anchor-based routing, with token-wise anchor selection and soft logit fusion enabling adaptive integration during decoding. LISA is fully plug-and-play and can be seamlessly integrated into existing MLLMs, including Qwen2.5-VL. Experiments on multiple benchmarks show that LISA reduces hallucinations by up to 53.6% in $\text{CHAIR}_\text{I}$ and improves POPE F1 by up to 5.1%, demonstrating strong generalization across models and tasks. Our code is available at https://github.com/zhlisa1010-eng/LISA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
