面向实时程序性视频理解的内存高效流式视频大语言模型
Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding
AI总结：
提出ProVideLLM，一种内存高效的流式视频LLM，通过多模态缓存减少令牌数22倍，实现10FPS推理和25FPS对话，并在六个程序性任务上达到SOTA。
摘要：
我们提出了ProVideLLM，一个用于实时程序性视频理解的端到端框架。ProVideLLM集成了一个多模态缓存，用于存储两种类型的令牌——言语化文本令牌，提供长期观察的压缩文本摘要，以及视觉令牌，使用DETR-QFormer编码以捕获短期观察的细粒度细节。这种设计在表示一小时的长期观察时，将令牌数量比现有方法减少了22倍，同时有效编码当前时刻的细粒度。通过在多模态缓存中交错这些令牌，ProVideLLM确保内存和计算随视频长度呈次线性扩展，实现每秒10帧的逐帧流式推理和每秒25帧的流式对话，且仅需2GB的GPU内存占用。ProVideLLM还在四个数据集的六个程序性任务上创下了新的最先进结果。
英文摘要：
We introduce ProVideLLM, an end-to-end framework for real-time procedural video understanding. ProVideLLM integrates a multimodal cache configured to store two types of tokens - verbalized text tokens, which provide compressed textual summaries of long-term observations, and visual tokens, encoded with DETR-QFormer to capture fine-grained details from short-term observations. This design reduces token count by 22x over existing methods in representing one hour of long-term observations while effectively encoding fine-granularity of the present. By interleaving these tokens in our multimodal cache, ProVideLLM ensures sub-linear scaling of memory and compute with video length, enabling per-frame streaming inference at 10 FPS and streaming dialogue at 25 FPS, with a minimal 2GB GPU memory footprint. ProVideLLM also sets new state-of-the-art results on six procedural tasks across four datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
