Eve：具有弹性视觉专家模型的高效多模态视觉语言模型
Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts
AI总结：
针对边缘设备上VLM应用挑战，提出Eve框架，通过多阶段弹性视觉专家整合平衡语言与多模态能力，仅1.8B参数即在语言和VLM基准上超越更大模型，取得68.87%最优结果。
摘要：
多模态视觉语言模型（VLM）在模型规模和数据集不断增长的支撑下取得了显著进展。然而，在边缘设备上运行VLM已成为其广泛应用的一大挑战。目前已有若干高效VLM的尝试，但它们往往牺牲语言能力以增强多模态能力，或者需要大量的训练。为解决这一困境，我们提出了创新框架——具有弹性视觉专家模型的高效视觉语言模型（Eve）。通过在训练的多个阶段策略性地整合可适应的视觉专业知识，Eve在保持语言能力与增强多模态能力之间取得了平衡。这种平衡的方法产生了一个仅有18亿参数的通用模型，在多模态和语言任务上均带来了显著提升。值得注意的是，在低于30亿参数的配置中，Eve在语言基准测试中表现明显优于其他模型，并在VLM基准测试中取得了68.87%的最新最优结果。此外，其多模态准确率超过了规模更大的70亿参数的LLaVA-1.5模型。我们的代码可在https://github.com/rangmiao/Eve获取。
英文摘要：
Multimodal vision language models (VLMs) have made significant progress with the support of continuously increasing model sizes and data volumes. Running VLMs on edge devices has become a challenge for their widespread application. There are several efficient VLM efforts, but they often sacrifice linguistic capabilities to enhance multimodal abilities, or require extensive training. To address this quandary,we introduce the innovative framework of Efficient Vision Language Models with Elastic Visual Experts (Eve). By strategically incorporating adaptable visual expertise at multiple stages of training, Eve strikes a balance between preserving linguistic abilities and augmenting multimodal capabilities. This balanced approach results in a versatile model with only 1.8B parameters that delivers significant improvements in both multimodal and linguistic tasks. Notably, in configurations below 3B parameters, Eve distinctly outperforms in language benchmarks and achieves state-of-the-art results 68.87% in VLM Benchmarks. Additionally, its multimodal accuracy outstrips that of the larger 7B LLaVA-1.5 model. Our code is available at https://github.com/rangmiao/Eve.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
