arXiv 2503.062202025-09-07cs.CVcs.LG

StreamMind：通过事件门控认知实现全帧率流式视频对话

StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition

University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）；Nanjing University（南京大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）

Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Donglin Bai, Zhibo Chen, Ting Cao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

StreamMind提出事件门控LLM调用范式，通过认知门控网络和事件保持特征提取器，实现全帧率流式视频对话，在Ego4D和SoccerNet上达到最先进性能。

摘要：

随着现实世界中人类与人工智能交互应用（如AI助手）的兴起，对流式视频对话的需求变得至关重要。为满足这一需求，我们提出了StreamMind，一个视频LLM框架，实现了超帧率流式视频处理（在单个A100上达到100 fps），并能够实时提供主动的、始终在线的响应，无需用户明确干预。为解决线性视频流速度与二次方Transformer计算成本之间的矛盾这一关键挑战，我们提出了一种新颖的感知-认知交错范式，称为“事件门控LLM调用”，与现有的每时间步LLM调用形成对比。通过在视频编码器和LLM之间引入认知门控网络，仅在相关事件发生时调用LLM。为实现恒定成本的事件特征提取，我们提出了基于状态空间方法的事件保持特征提取器（EPFE），为时空特征生成单一感知令牌。这些技术使视频LLM具备全帧率感知和实时认知响应能力。在Ego4D和SoccerNet流式任务以及标准离线基准上的实验表明，在模型能力和实时效率方面均达到了最先进水平，为游戏AI和交互媒体等超高帧率应用铺平了道路。代码和数据可在https://aka.ms/StreamMind获取。

英文摘要：

With the rise of real-world human-AI interaction applications, such as AI assistants, the need for Streaming Video Dialogue is critical. To address this need, we introduce StreamMind, a video LLM framework that achieves ultra-FPS streaming video processing (100 fps on a single A100) and enables proactive, always-on responses in real time, without explicit user intervention. To solve the key challenge of the contradiction between linear video streaming speed and quadratic transformer computation cost, we propose a novel perception-cognition interleaving paradigm named ''event-gated LLM invocation'', in contrast to the existing per-time-step LLM invocation. By introducing a Cognition Gate network between the video encoder and the LLM, LLM is only invoked when relevant events occur. To realize the event feature extraction with constant cost, we propose Event-Preserving Feature Extractor (EPFE) based on state-space method, generating a single perception token for spatiotemporal features. These techniques enable the video LLM with full-FPS perception and real-time cognition response. Experiments on Ego4D and SoccerNet streaming tasks, as well as standard offline benchmarks, demonstrate state-of-the-art performance in both model capability and real-time efficiency, paving the way for ultra-high-FPS applications, such as Game AI and interactive media. The code and data is available at https://aka.ms/StreamMind.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。