基于事件先验的视觉-语言模型用于高效视觉理解
Event-Priori-Based Vision-Language Model for Efficient Visual Understanding
AI总结：
本文提出 EP-VLM，利用事件视觉运动先验引导 RGB patch 稀疏化并保持位置编码，在 Qwen2-VL-2B 上节省 50% FLOPs 且保留 98% 精度。
摘要：
基于大语言模型（LLM）的视觉-语言模型（VLM）已大幅拓展了视觉理解能力的边界。然而，其高昂的计算需求阻碍了在资源受限边缘设备上的部署。低效的一个关键来源在于 VLM 需要处理密集且冗余的视觉信息。视觉输入包含大量与文本语义无关的区域，导致相关计算对推理无效。本文提出一种新颖的基于事件先验的视觉-语言模型，称为 EP-VLM。其核心贡献是一种利用来自动态事件视觉的运动先验来提升 VLM 效率的新机制。受人类视觉认知启发，EP-VLM 首先使用事件数据引导 RGB 视觉输入的按 patch 稀疏化，逐步将 VLM 计算集中到视觉输入的显著区域。随后，我们为 VLM 架构中的视觉编码器构建了一种保持位置的分词策略。该策略处理事件引导的非结构化稀疏视觉输入，同时准确保持视觉输入内的位置理解。实验结果表明，与 Qwen2-VL 系列的基线模型相比，EP-VLM 在保持近乎无损精度的同时实现了显著的效率提升。例如，相较于原始 Qwen2-VL-2B，EP-VLM 实现了 50% 的 FLOPs 节省，同时在 RealWorldQA 数据集上保留了 98% 的原始精度。本工作展示了基于事件的视觉先验在提升 VLM 推理效率方面的潜力，为创建更高效、可部署的 VLM、在边缘实现可持续视觉理解铺平了道路。
英文摘要：
Large Language Model (LLM)-based Vision-Language Models (VLMs) have substantially extended the boundaries of visual understanding capabilities. However, their high computational demands hinder deployment on resource-constrained edge devices. A key source of inefficiency stems from the VLM's need to process dense and redundant visual information. Visual inputs contain significant regions irrelevant to text semantics, rendering the associated computations ineffective for inference. This paper introduces a novel Event-Priori-Based Vision-Language Model, termed EP-VLM. Its core contribution is a novel mechanism leveraging motion priors derived from dynamic event vision to enhance VLM efficiency. Inspired by human visual cognition, EP-VLM first employs event data to guide the patch-wise sparsification of RGB visual inputs, progressively concentrating VLM computation on salient regions of the visual input. Subsequently, we construct a position-preserving tokenization strategy for the visual encoder within the VLM architecture. This strategy processes the event-guided, unstructured, sparse visual input while accurately preserving positional understanding within the visual input. Experimental results demonstrate that EP-VLM achieves significant efficiency improvements while maintaining nearly lossless accuracy compared to baseline models from the Qwen2-VL series. For instance, against the original Qwen2-VL-2B, EP-VLM achieves 50% FLOPs savings while retaining 98% of the original accuracy on the RealWorldQA dataset. This work demonstrates the potential of event-based vision priors for improving VLM inference efficiency, paving the way for creating more efficient and deployable VLMs for sustainable visual understanding at the edge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
