VideoLLM-MoD：基于深度混合视觉计算的高效视频语言流处理
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation
AI总结：
VideoLLM-MoD通过让冗余视觉token跳过transformer层计算（如80%），在保持或提升性能的同时显著降低训练的内存与时间开销，有效解决长视频流处理中的计算瓶颈。
摘要：
大型视觉语言模型（如GPT-4、LLaVA）面临一个众所周知的困境：虽然增加视觉token的数量通常能增强视觉理解能力，但也会显著增加内存和计算成本，这在长期、密集的视频帧流处理场景中尤为明显。尽管已经开发了Q-Former和Perceiver Resampler等可学习方法来减轻视觉token负担，但它们忽略了LLM因果建模的上下文（即键值缓存），在处理用户查询时可能导致遗漏视觉线索。在本文中，我们引入一种新方法，通过利用冗余视觉token“跳过层”来减少视觉计算，而不是减少视觉token的数量。我们的方法VideoLLM-MoD受深度混合LLM启发，旨在解决长期或流式视频中大量视觉token的挑战。具体而言，对于每个transformer层，我们学习跳过高比例（例如80%）视觉token的计算，将其直接传递到下一层。这种方法显著提高了模型效率，在整个训练过程中实现了约42%的时间和约30%的内存节省。此外，我们的方法减少了上下文中的计算并避免减少视觉token，从而与原始模型相比保持甚至提高了性能。我们进行了大量实验以证明VideoLLM-MoD的有效性，展示了其在多个基准测试上的最优结果，包括COIN、Ego4D和Ego-Exo4D数据集中的叙述、预测和总结任务。
英文摘要：
A well-known dilemma in large vision-language models (e.g., GPT-4, LLaVA) is that while increasing the number of vision tokens generally enhances visual understanding, it also significantly raises memory and computational costs, especially in long-term, dense video frame streaming scenarios. Although learnable approaches like Q-Former and Perceiver Resampler have been developed to reduce the vision token burden, they overlook the context causally modeled by LLMs (i.e., key-value cache), potentially leading to missed visual cues when addressing user queries. In this paper, we introduce a novel approach to reduce vision compute by leveraging redundant vision tokens "skipping layers" rather than decreasing the number of vision tokens. Our method, VideoLLM-MoD, is inspired by mixture-of-depths LLMs and addresses the challenge of numerous vision tokens in long-term or streaming video. Specifically, for each transformer layer, we learn to skip the computation for a high proportion (e.g., 80\%) of vision tokens, passing them directly to the next layer. This approach significantly enhances model efficiency, achieving approximately \textasciitilde42\% time and \textasciitilde30\% memory savings for the entire training. Moreover, our method reduces the computation in the context and avoid decreasing the vision tokens, thus preserving or even improving performance compared to the vanilla model. We conduct extensive experiments to demonstrate the effectiveness of VideoLLM-MoD, showing its state-of-the-art results on multiple benchmarks, including narration, forecasting, and summarization tasks in COIN, Ego4D, and Ego-Exo4D datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
