StreamingAssistant: 为加速在线视频理解的高效视觉标记修剪
StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding
AI总结：
StreamingAssistant通过高效视觉标记修剪技术，提升在线视频理解的准确性和效率，减少GPU内存使用和计算延迟。
摘要：
在线视频理解对于公共监控和人工智能眼镜等应用至关重要。然而，将多模态大语言模型（MLLMs）应用于该领域具有挑战性，因为视频帧数量庞大，导致GPU内存使用高且计算延迟大。为解决这些问题，我们提出标记修剪作为一种减少上下文长度同时保留关键信息的方法。具体来说，我们引入了一个新的冗余度度量标准，即最大相似度到空间相邻视频标记（MSSAVT），该度量标准考虑了标记相似性和空间位置。为缓解修剪与冗余之间的双向依赖性，我们进一步设计了一种掩码修剪策略，确保只修剪互不相邻的标记。我们还集成了一个现有的基于时间冗余的修剪方法，以消除视频模态的时间冗余。在多个在线和离线视频理解基准测试中的实验结果表明，我们的方法显著提高了准确性（即最多提高4%），同时引入了可忽略的修剪延迟（即不到1毫秒）。我们的完整实现将公开发布。
英文摘要：
Online video understanding is essential for applications like public surveillance and AI glasses. However, applying Multimodal Large Language Models (MLLMs) to this domain is challenging due to the large number of video frames, resulting in high GPU memory usage and computational latency. To address these challenges, we propose token pruning as a means to reduce context length while retaining critical information. Specifically, we introduce a novel redundancy metric, Maximum Similarity to Spatially Adjacent Video Tokens (MSSAVT), which accounts for both token similarity and spatial position. To mitigate the bidirectional dependency between pruning and redundancy, we further design a masked pruning strategy that ensures only mutually unadjacent tokens are pruned. We also integrate an existing temporal redundancy-based pruning method to eliminate temporal redundancy of the video modality. Experimental results on multiple online and offline video understanding benchmarks demonstrate that our method significantly improves the accuracy (i.e., by 4\% at most) while incurring a negligible pruning latency (i.e., less than 1ms). Our full implementation will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
