动态多模态激活引导用于大型视觉-语言模型中的幻觉缓解
Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
AI总结：
本文提出动态多模态激活引导方法，通过构建语义真实性引导向量数据库，实现幻觉缓解，提升模型性能。
摘要：
大型视觉-语言模型（LVLMs）在视觉-语言任务上表现出色，但在幻觉问题上存在困难。通过对LVLM激活模式的深入分析，我们揭示了两个关键发现：1）真实性与视觉感知能力主要在模型架构中激活不同的注意力头子集；2）真实性引导向量在不同的语义上下文中变化显著。基于这些观察，我们提出动态多模态激活引导，一种无需训练的幻觉缓解方法。我们的方法构建了一个基于语义的真实性引导向量数据库，并计算视觉感知引导向量，通过在推理过程中动态选择最相关的引导向量，基于输入语义相似性并应用于最影响性的注意力头，实现上下文感知干预。我们进行了跨多个模型和数据集的全面实验，证明我们的方法显著提高了模型性能，优于现有最先进的方法。
英文摘要：
Large Vision-Language Models (LVLMs) exhibit outstanding performance on vision-language tasks but struggle with hallucination problems. Through in-depth analysis of LVLM activation patterns, we reveal two key findings: 1) truthfulness and visual perception capabilities predominantly engage different subsets of attention heads within the model architecture; and 2) truthfulness steering vectors vary significantly across different semantic contexts. Based on these observations, we propose Dynamic Multimodal Activation Steering, a training-free approach for hallucination mitigation. Our method constructs a semantic-based truthfulness steering vector database and computes visual perception steering vectors, enabling context-aware interventions during inference by dynamically selecting the most relevant steering vectors based on input semantic similarity and applying them to the most influential attention heads. We conduct comprehensive experiments across multiple models and datasets, demonstrating that our approach significantly enhances model performance, outperforming existing state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
