Dispider：通过解耦感知、决策与反应使视频大语言模型具备主动实时交互能力
Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction
AI总结：
本文提出 Dispider，通过解耦并异步执行感知、决策与反应，使视频大语言模型能在长时流式视频中主动、实时且高效地交互。
摘要：
与视频大语言模型（Video LLMs）进行主动实时交互引入了一种人机交互新范式，模型不仅要理解用户意图，还要在持续即时处理流式视频的同时作出响应。不同于在回答问题前先分析整段视频的离线视频大语言模型，主动实时交互需要三项能力：1）感知：实时视频监控与交互捕捉；2）决策：在适当情境下发起主动交互；3）反应：与用户持续交互。然而，这些期望能力之间存在内在冲突。决策和反应需要相反的感知尺度与粒度，并且在反应期间，自回归解码会阻塞实时感知和决策。为了在一个协调系统中统一这些相互冲突的能力，我们提出了 Dispider，这是一个解耦感知、决策与反应的系统。Dispider 具有轻量级主动流式视频处理模块，可跟踪视频流并识别最佳交互时刻。一旦交互被触发，异步交互模块会提供详细响应，同时处理模块在此期间继续监控视频。我们的解耦且异步设计确保响应及时、上下文准确且计算高效，使 Dispider 非常适合长时视频流的主动实时交互。实验表明，Dispider 不仅在传统视频问答任务中保持强劲性能，还在流式场景响应方面显著超越以往在线模型，从而验证了我们架构的有效性。代码和模型已发布于 https://github.com/Mark12Ding/Dispider。
英文摘要：
Active Real-time interaction with video LLMs introduces a new paradigm for human-computer interaction, where the model not only understands user intent but also responds while continuously processing streaming video on the fly. Unlike offline video LLMs, which analyze the entire video before answering questions, active real-time interaction requires three capabilities: 1) Perception: real-time video monitoring and interaction capturing. 2) Decision: raising proactive interaction in proper situations, 3) Reaction: continuous interaction with users. However, inherent conflicts exist among the desired capabilities. The Decision and Reaction require a contrary Perception scale and grain, and the autoregressive decoding blocks the real-time Perception and Decision during the Reaction. To unify the conflicted capabilities within a harmonious system, we present Dispider, a system that disentangles Perception, Decision, and Reaction. Dispider features a lightweight proactive streaming video processing module that tracks the video stream and identifies optimal moments for interaction. Once the interaction is triggered, an asynchronous interaction module provides detailed responses, while the processing module continues to monitor the video in the meantime. Our disentangled and asynchronous design ensures timely, contextually accurate, and computationally efficient responses, making Dispider ideal for active real-time interaction for long-duration video streams. Experiments show that Dispider not only maintains strong performance in conventional video QA tasks, but also significantly surpasses previous online models in streaming scenario responses, thereby validating the effectiveness of our architecture. The code and model are released at \url{https://github.com/Mark12Ding/Dispider}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
