发表机构
Nanjing University of Science and Technology; Ant Group; National University of Singapore; The Chinese University of Hong Kong（南京理工大学; 蚂蚁集团; 新加坡国立大学; 香港中文大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究针对流视频理解中存储-检索范式无法内化历史证据的问题，提出LatentStream框架，通过分层内存组织、渐进式演化及置信度优化实现流记忆的检索-内化，在视频基准上取得最优结果。
AI 中文摘要
流视频理解要求多模态大语言模型（MLLMs）处理连续视觉输入，并在严格因果性和有限内存约束下响应用户查询。现有方法通常将历史观测压缩到外部内存库中，检索与查询相关的证据作为额外视觉上下文。尽管有效，但这种存储-检索范式将历史证据保留为外部视觉上下文，无法将其内化到紧凑、可演化的潜在记忆中以持续指导流推理。为弥合这一差距，我们提出LatentStream，一种渐进式潜在工作记忆框架，将流内存从存储-检索范式转向检索-内化范式。具体而言，LatentStream包含三个协同组件：第一，查询无关分层流内存在固定内存预算下，通过Jenks引导的自适应整合将视觉历史组织为短期、中期和长期层级；第二，当查询到达时，分层潜在记忆演化为各组潜在记忆令牌配备逐步扩展的记忆感受野，使其能从对应范围迭代检索历史证据并将其内化为紧凑固定长度的潜在记忆；第三，渐进式置信度引导的潜在记忆优化从分组预测熵构建分层递进奖励，联合优化潜在记忆令牌和检索到的证据，以推动流推理置信度逐步提升。大量实验表明，LatentStream在现有在线和离线视频基准上达到了新的最优结果。
英文摘要
Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
