arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2603.01455cs.CVcs.AIcs.CLcs.IRcs.MM

从逐字到概要：基于语义信息瓶颈的金字塔多模态记忆压缩用于长视界视频智能体

From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents

Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳校区）；Peng Cheng Laboratory（鹏城实验室）

Niu Lian, Yuting Wang, Hanshu Yao, Jinpeng Wang, Bin Chen, Yaowei Wang, Min Zhang, Shu-Tao Xia

备注：
Accepted by ACL 2026 Main. 17 pages, 7 figures, 8 tables. TL;DR: We propose MM-Mem, a cognition-inspired, dual-trace hierarchical memory framework for long-horizon video understanding grounded in Fuzzy-Trace Theory. It features adaptive memory compression via the Information Bottleneck and employs an entropy-driven top-down retrieval to access fine-grained details only when necessary
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MM-Mem架构，通过金字塔多模态记忆压缩，结合语义信息瓶颈目标，实现长视界视频理解中的高效记忆组织与任务相关信息保留。

摘要：

尽管多模态大语言模型在短期推理上表现出色，但受限于有限的上下文窗口和静态内存机制，难以处理长视界视频理解。现有方法通常面临两种极端：视觉中心方法通过密集视觉积累导致高延迟和冗余，文本中心方法则因激进的标题生成而损失细节和幻觉。为此，本文提出MM-Mem，基于模糊痕迹理论，将记忆分层为感觉缓冲区、事件流和符号模式，逐步将细粒度感知痕迹（逐字）压缩为高层语义模式（概要）。此外，为指导内存的动态构建，推导出语义信息瓶颈目标，并引入SIB-GRPO优化内存压缩与任务相关信息保留的平衡。在推理中，设计了基于熵驱动的自顶向下的内存检索策略。在四个基准测试中，MM-Mem在离线和流式任务中均取得最佳性能，证明了认知启发式记忆组织的有效性。代码和相关配置可在https://github.com/EliSpectre/MM-Mem公开获取。

英文摘要：

While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy. Extensive experiments across 4 benchmarks confirm that MM-Mem achieves state-of-the-art performance on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code and associated configurations are publicly available at https://github.com/EliSpectre/MM-Mem.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑