arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.07512cs.CVcs.AIcs.CL

MemDreamer: 通过分层图记忆和智能体检索机制解耦感知与推理以实现长视频理解

MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism

Ant Group(蚂蚁集团);Zhejiang University(浙江大学);Central South University(中南大学);HKUST(GZ)(香港科技大学(广州))

Cong Chen, Guo Gan, Kaixiang Ji, ZhaoYang Zhang, Zhen Yang, Guangming Yao, Hao Chen, Jingdong Chen, Yi Yuan, Chunhua Shen

AI总结:

提出MemDreamer框架,通过分层图记忆和智能体检索机制解耦感知与推理,将长视频理解转化为智能体探索过程,在四个基准上达到SOTA,推理上下文窗口仅占全量2%且准确率提升12.5点。

摘要:

当前的视觉-语言模型在处理数小时长的视频时面临困难,因为处理完整长度的视觉序列会导致令牌爆炸和注意力稀释。为了克服这一问题,我们引入了MemDreamer,将感知与推理解耦,将长视频理解转化为智能体探索过程。作为一个即插即用的框架,它增量式地流式传输视频以构建分层图记忆,这是一种自顶向下的三层架构,用于语义抽象,并由一个捕获时空和因果关系的基础图锚定。在推理过程中,推理模型采用智能体工具增强的检索,通过观察-推理-行动循环导航层次结构、搜索节点和遍历逻辑边。实验表明,MemDreamer在四个主流基准上取得了最先进的结果,将人类专家的差距缩小到仅3.7个百分点。它将推理上下文窗口限制在全量上下文的仅2%,同时提供了12.5个百分点的绝对准确率提升。此外,统计分析揭示了VLM在逻辑推理和长视频理解基准上的性能之间存在强正线性相关,将智能体能力扩展确立为多模态理解的新范式。

英文摘要:

Current Vision-Language Models struggle with hours-long videos because processing full-length visual sequences induces prohibitive token explosion and attention dilution. To overcome this, we introduce MemDreamer to decouple perception and reasoning, shifting long-video understanding into an agentic exploration process. As a plug-and-play framework, it incrementally streams videos to construct a Hierarchical Graph Memory, a top-down three-tier architecture for semantic abstraction, anchored by a foundational graph capturing spatiotemporal and causal relations. During inference, the reasoning model employs agentic tool-augmented retrieval, navigating hierarchies, searching nodes, and traversing logical edges via an Observation-Reason-Action loop. Experiments show MemDreamer achieves SOTA results across four mainstream benchmarks, narrowing the gap with human experts to only 3.7 points. It constrains the reasoning context window to merely 2% of full-context ingestion while delivering a 12.5 point absolute accuracy gain. Furthermore, statistical analysis uncovers a strong positive linear correlation between an VLM's performance on logic reasoning and long-video understanding benchmarks, establishing agentic capability scaling as a new paradigm for multimodal comprehension.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑