扩展多模态大语言模型中视觉先验知识的边界
Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models
AI总结：
研究提出度量指标$Rank_e$量化视觉编码器先验知识对多模态大语言模型性能的影响，并设计两阶段训练框架VisPRE显式增强视觉先验知识，显著提升了模型对不常见视觉实体的理解能力。
摘要：
视觉编码器的先验知识是否制约了多模态大语言模型（MLLMs）的能力边界？尽管大多数现有研究将MLLMs视为通过端到端训练优化的统一系统，但视觉编码器先验知识的影响却鲜少被研究。在这项工作中，我们引入了一种新的度量指标$Rank_e$，以量化视觉编码器先验知识对MLLM性能的影响。我们的分析揭示了先验知识与MLLM性能之间存在正相关关系。此外，我们发现仅使用端到端视觉问答（VQA）数据进行领域特定微调是不充分的，特别是对于具有较低内在视觉先验知识的实体。为解决这一问题，我们提出了VisPRE（视觉先验修复），这是一个在视觉编码器层面显式引入先验知识的两阶段训练框架。实验结果表明，增强视觉编码器的先验知识能大幅提升MLLMs的视觉理解能力，为改善性能提供了一种新颖且有效的策略，尤其是在涉及不常见视觉实体的场景中。
英文摘要：
Does the prior knowledge of the vision encoder constrain the capability boundary of Multi-modal Large Language Models (MLLMs)? While most existing research treats MLLMs as unified systems optimized through end-to-end training, the impact of vision encoder's prior knowledge is seldom investigated. In this work, we introduce a novel metric, $Rank_e$, to quantify the effect of prior knowledge of the vision encoder on MLLM performance. Our analysis reveals a positive correlation between prior knowledge and MLLM performance. Moreover, we find that domain-specific fine-tuning using solely end-to-end visual question answering (VQA) data is insufficient, particularly for entities with low inherent visual prior knowledge. To address this issue, we propose VisPRE (Vision Prior Remediation), a two-stage training framework that explicitly incorporates prior knowledge at the vision encoder level. Experimental results demonstrate that augmenting vision encoder's prior knowledge substantially boosts the visual understanding capabilities of MLLMs, offering a novel and effective strategy for improving performance, especially in scenarios involving uncommon visual entities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
