arXiv 2412.012892024-12-04cs.CVcs.AI

通过无需训练的融合增强多模态大语言模型的感知能力

Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

ZIP Lab, Zhejiang University（浙江大学 ZIP 实验室）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人教育部重点实验室）

Zhuokun Chen, Jinwu Hu, Zeshuai Deng, Yufeng Wang, Bohan Zhuang, Mingkui Tan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有增强多模态大语言模型视觉感知需高成本训练的问题，提出无需训练的VisionFuse框架，通过融合同家族现成MLLMs的视觉编码器token并合并语言模型参数，显著降低部署开销，在多模态基准上实现超4%的性能提升。

摘要：

多模态大语言模型（MLLMs）通过将视觉编码器与语言模型对齐，为语言模型赋予视觉能力。现有增强MLLMs视觉感知的方法通常涉及设计更强大的视觉编码器，这需要探索庞大的设计空间，并将每个潜在编码器与语言模型重新对齐，导致训练成本高得令人望而却步。在本文中，我们提出VisionFuse，一种新颖的集成框架，无需额外训练即可高效利用现成MLLMs中的多个视觉编码器来增强视觉感知。我们的方法源于观察：不同MLLMs在给定相同查询和图像时，往往关注不同区域。此外，我们发现同一MLLM家族（共享相同预训练LLM的一组MLLMs）内的视觉编码器特征分布高度对齐。基于这些见解，VisionFuse通过连接所选家族内MLLMs视觉编码器生成的token来丰富视觉上下文。通过合并这些MLLMs的语言模型参数，VisionFuse允许单个语言模型与多种视觉编码器对齐，显著降低部署开销。我们使用各种MLLM组合在多个多模态基准上进行综合评估，证明在多模态任务中取得显著改进。值得注意的是，当集成MiniGemini-8B和SLIME-8B时，VisionFuse实现了超过4%的平均性能提升。

英文摘要：

Multimodal LLMs (MLLMs) equip language models with visual capabilities by aligning vision encoders with language models. Existing methods to enhance the visual perception of MLLMs often involve designing more powerful vision encoders, which requires exploring a vast design space and re-aligning each potential encoder with the language model, resulting in prohibitively high training costs. In this paper, we introduce VisionFuse, a novel integration framework that efficiently utilizes multiple vision encoders from off-the-shelf MLLMs to enhance visual perception without requiring additional training. Our approach is motivated by the observation that different MLLMs tend to focus on distinct regions given the same query and image. Moreover, we find that the feature distributions of vision encoders within an MLLM family, a group of MLLMs sharing the same pretrained LLM, are highly aligned. Building on these insights, VisionFuse enriches the visual context by concatenating the tokens generated by the vision encoders of selected MLLMs within a family. By merging the parameters of language models from these MLLMs, VisionFuse allows a single language model to align with various vision encoders, significantly reducing deployment overhead. We conduct comprehensive evaluations across multiple multimodal benchmarks using various MLLM combinations, demonstrating substantial improvements in multimodal tasks. Notably, when integrating MiniGemini-8B and SLIME-8B, VisionFuse achieves an average performance increase of over 4%.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。