EVLM：用于视觉理解的高效视觉-语言模型
EVLM: An Efficient Vision-Language Model for Visual Understanding
AI总结：
本文提出EVLM，一种高效视觉-语言模型，通过交叉注意力、分层ViT特征和专家混合机制，在降低计算成本的同时全面感知视觉信号，并在多模态基准和字幕生成任务上取得良好表现。
摘要：
在多模态语言模型领域，大多数方法基于类似于LLaVA的架构构建。这些模型使用单层ViT特征作为视觉提示，将其与文本标记一起直接输入到语言模型中。然而，在处理长序列视觉信号或视频等输入时，语言模型的自注意力机制会导致显著的计算开销。此外，使用单层ViT特征使得大型语言模型难以全面感知视觉信号。本文提出了一种高效的多模态语言模型，以在尽可能全面地感知视觉信号的同时最小化计算成本。我们的方法主要包括：（1）采用类似于Flamingo的交叉注意力进行图像-文本交互。（2）利用分层ViT特征。（3）引入专家混合（MoE）机制以增强模型效果。我们的模型在公共多模态基准测试上取得了具有竞争力的分数，并在图像字幕生成和视频字幕生成等任务中表现良好。
英文摘要：
In the field of multi-modal language models, the majority of methods are built on an architecture similar to LLaVA. These models use a single-layer ViT feature as a visual prompt, directly feeding it into the language models alongside textual tokens. However, when dealing with long sequences of visual signals or inputs such as videos, the self-attention mechanism of language models can lead to significant computational overhead. Additionally, using single-layer ViT features makes it challenging for large language models to perceive visual signals fully. This paper proposes an efficient multi-modal language model to minimize computational costs while enabling the model to perceive visual signals as comprehensively as possible. Our method primarily includes: (1) employing cross-attention to image-text interaction similar to Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of Experts (MoE) mechanism to enhance model effectiveness. Our model achieves competitive scores on public multi-modal benchmarks and performs well in tasks such as image captioning and video captioning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
