不要暂停：面向在线视频理解的流式视频-语言同步
Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding
AI总结：
提出流式视频-语言同步（SVLS）范式，通过帧驱动转换控制器和流式令牌调节器实现视频帧与语言生成的细粒度同步，在不中断感知的情况下进行实时交互。
摘要：
在线视频大语言模型（Video-LLMs）通过逐帧处理和主动响应，在人机交互方面取得了进展。然而，流式场景中仍存在一个关键挑战：现有模型在生成响应时通常会暂停视频感知，破坏了实时的视频-语言同步并导致卡顿。为了解决这个问题，我们引入了一种新的在线视频理解范式：流式视频-语言同步（SVLS），并提出了LyraV，一个基于分层控制框架的实时流式助手，具有两个核心创新。首先，帧驱动转换控制器（FDTC）是一个无需训练的基于验证的有限状态机，它做出高层语义决策，决定何时继续说话、开始新的响应或保持沉默。其次，流式令牌调节器（SToP）是一个即插即用的轻量级预测模块，动态调整语言生成速率以匹配视觉内容的节奏。具体来说，LyraV执行逐帧增量、子预算解码：在每个帧间隔内，它只发射适合实时预算的一小部分令牌，因此感知永远不会被阻塞整个句子。这些组件共同使LyraV能够无缝地交织传入的视频帧和生成的词令牌，实现细粒度的同步。在五个在线和三个离线基准上进行的广泛实验表明，LyraV保留了骨干网络的通用理解能力，同时显著提高了流式同步和叙事流畅性，实现了98.29%的视频播放同步率和3.89 FPS的实时处理速度。有趣的是，我们观察到LyraV的一个经验能力：对流式令牌进行动态推理，实现了与视觉输入并行的连续解释和“思考”。
英文摘要：
Online Video Large Language Models (Video-LLMs) have advanced toward seamless human-AI interaction through frame-by-frame processing and proactive responding. However, a critical challenge remains in streaming scenarios: existing models typically pause video perception while generating responses, breaking real-time video-language synchrony and causing stutters. To address this, we introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS), and present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations. First, the Frame-Driven Transition Controller (FDTC), a training-free verification-based finite-state machine, makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent. Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content. Concretely, LyraV performs \emph{per-frame incremental, sub-budget decoding}: within each frame interval it emits only a small chunk of tokens that fits the real-time budget, so perception is never blocked for a full sentence. Together, these components enable LyraV to seamlessly interleave incoming video frames with generated word tokens, achieving a fine-grained synchrony. Extensive experiments conducted on five online and three offline benchmarks demonstrate that LyraV preserves the backbone's general understanding ability while substantially improving streaming synchrony and narrative fluency, delivering a 98.29\% synchrony with video playback and a real-time processing speed of 3.89 FPS. Interestingly, we observe an empirical capability in LyraV: dynamic reasoning over streaming tokens, enabling continuous interpretation and "thinking" alongside visual input.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
