从视觉视角探究缓解大型视觉语言模型的幻觉问题
Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models
AI总结：
针对大型视觉语言模型的物体幻觉问题，提出包含约一万样本的细粒度基准VHBench-10，并设计上下文感知路由网络VisionWeaver动态聚合多专家视觉特征，有效减少幻觉并提升性能。
摘要：
大型视觉语言模型（LVLMs）中的物体幻觉严重阻碍了其在现实世界中的适用性。作为准确解释视觉信息的主要组件，视觉编码器的选择至关重要。我们假设，不同视觉编码器采用的多样化训练范式赋予其独特的归纳偏置，从而导致其各异的幻觉表现。现有基准通常侧重于粗粒度的幻觉检测，无法捕捉我们假设中所详述的多样化幻觉。为系统分析这些影响，我们引入了VHBench-10，这是一个包含约10,000个样本的综合基准，用于在十个细粒度幻觉类别上评估LVLMs。我们的评估证实编码器表现出独特的幻觉特征。基于这些见解以及简单特征融合的次优性，我们提出了VisionWeaver，一种新颖的上下文感知路由网络。它利用全局视觉特征生成路由信号，动态聚合来自多个专业化专家的视觉特征。综合实验证实了VisionWeaver在显著减少幻觉和提升整体模型性能方面的有效性。
英文摘要：
Object hallucination in Large Vision-Language Models (LVLMs) significantly impedes their real-world applicability. As the primary component for accurately interpreting visual information, the choice of visual encoder is pivotal. We hypothesize that the diverse training paradigms employed by different visual encoders instill them with distinct inductive biases, which leads to their diverse hallucination performances. Existing benchmarks typically focus on coarse-grained hallucination detection and fail to capture the diverse hallucinations elaborated in our hypothesis. To systematically analyze these effects, we introduce VHBench-10, a comprehensive benchmark with approximately 10,000 samples for evaluating LVLMs across ten fine-grained hallucination categories. Our evaluations confirm encoders exhibit unique hallucination characteristics. Building on these insights and the suboptimality of simple feature fusion, we propose VisionWeaver, a novel Context-Aware Routing Network. It employs global visual features to generate routing signals, dynamically aggregating visual features from multiple specialized experts. Comprehensive experiments confirm the effectiveness of VisionWeaver in significantly reducing hallucinations and improving overall model performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
