MoVA：将视觉专家混合模型适配至多模态上下文
MoVA: Adapting Mixture of Vision Experts to Multimodal Context
AI总结：
针对单一视觉编码器存在任务偏差的问题，提出新型多模态大语言模型MoVA，通过由粗到细的机制自适应路由融合多视觉专家，在多类多模态基准上取得显著性能提升。
摘要：
作为多模态大语言模型（MLLMs）的核心组件，视觉编码器的能力极大影响着MLLM对多样化图像内容的理解效果。尽管CLIP、DINOv2中的视觉编码器等大规模预训练视觉编码器已带来可观性能，但我们发现仍不存在单一视觉编码器能在各类图像内容理解任务中占据主导：例如CLIP视觉编码器在通用图像理解上表现优异，但在文档或图表内容上性能不佳。为缓解CLIP视觉编码器的偏差问题，我们首先深入探究了不同预训练视觉编码器的固有特性，随后提出MoVA——一种性能强大的新型MLLM，它通过由粗到细的机制自适应地路由并融合面向特定任务的视觉专家。在粗粒度阶段，我们设计了一种上下文感知的专家路由策略，可根据用户指令、输入图像以及视觉专家的专长动态选择最合适的视觉专家，这得益于大语言模型（LLM）强大的模型功能理解能力。在细粒度阶段，我们精心构建了视觉专家混合适配器（MoV-Adapter），用于从各类专家中提取并融合特定任务知识。这种由粗到细的范式能基于多模态上下文与模型专长有效利用各专家的表征，进一步提升泛化能力。我们开展了大量实验以评估所提方法的有效性，在未添加任何额外技巧的情况下，MoVA在一系列具有挑战性的多模态基准测试中，相较当前最优方法实现了显著的性能提升。
英文摘要：
As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM). In the fine-grained stage, we elaborately conduct the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fine paradigm effectively leverages representations from experts based on multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments to evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging multimodal benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
