arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.12422cs.CV
收藏

VideoLucy:用于长视频理解的深度记忆回溯

VideoLucy: Deep Memory Backtracking for Long Video Understanding

National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(多谱段信息智能处理国家级重点实验室,人工智能与自动化学院,华中科技大学);NUS(国立大学新加坡);S-Lab, NTU(NTU的S实验室);Shanghai AI Lab(上海人工智能实验室)

Jialong Zuo, Yongtai Deng, Lingdong Kong, Jingkang Yang, Rui Jin, Yiwei Zhang, Nong Sang, Liang Pan, Ziwei Liu, Changxin Gao

AI总结:

提出VideoLucy框架,采用层次记忆结构与智能体迭代回溯机制,解决长视频理解中时间上下文缺失和稀疏采样丢信息问题,并在新基准EgoMem上超越现有方法。

摘要:

近期研究表明,基于智能体的系统利用大型语言模型(LLM)进行关键信息检索与整合,已成为长视频理解的一种有前景的方法。然而,这些系统面临两大挑战。首先,它们通常对单个帧进行建模和推理,难以捕捉连续帧的时间上下文。其次,为降低密集帧级字幕生成的成本,它们采用稀疏帧采样,这有丢失关键信息的风险。为克服这些局限,我们提出VideoLucy,一个用于长视频理解的深度记忆回溯框架。受人类从粗略到精细的回忆过程启发,VideoLucy采用具有渐进粒度的层次记忆结构。该结构在不同层次深度上明确界定了记忆的细节级别和时间范围。通过基于智能体的迭代回溯机制,VideoLucy系统地挖掘视频范围内、与问题相关的深度记忆,直至收集到足够信息以给出可信答案。这一设计使得在保留关键细节的同时,能够有效理解连续帧的时间上下文。此外,我们引入EgoMem,一个用于长视频理解的新基准。EgoMem旨在全面评估模型理解随时间展开的复杂事件以及在极长视频中捕捉细粒度细节的能力。大量实验证明了VideoLucy的优越性。基于开源模型构建的VideoLucy在多个长视频理解基准上显著优于最先进的方法,其性能甚至超越了如GPT-4o等最新的专有模型。我们的代码和数据集将在https://videolucy.github.io公开提供。

英文摘要:

Recent studies have shown that agent-based systems leveraging large language models (LLMs) for key information retrieval and integration have emerged as a promising approach for long video understanding. However, these systems face two major challenges. First, they typically perform modeling and reasoning on individual frames, struggling to capture the temporal context of consecutive frames. Second, to reduce the cost of dense frame-level captioning, they adopt sparse frame sampling, which risks discarding crucial information. To overcome these limitations, we propose VideoLucy, a deep memory backtracking framework for long video understanding. Inspired by the human recollection process from coarse to fine, VideoLucy employs a hierarchical memory structure with progressive granularity. This structure explicitly defines the detail level and temporal scope of memory at different hierarchical depths. Through an agent-based iterative backtracking mechanism, VideoLucy systematically mines video-wide, question-relevant deep memories until sufficient information is gathered to provide a confident answer. This design enables effective temporal understanding of consecutive frames while preserving critical details. In addition, we introduce EgoMem, a new benchmark for long video understanding. EgoMem is designed to comprehensively evaluate a model's ability to understand complex events that unfold over time and capture fine-grained details in extremely long videos. Extensive experiments demonstrate the superiority of VideoLucy. Built on open-source models, VideoLucy significantly outperforms state-of-the-art methods on multiple long video understanding benchmarks, achieving performance even surpassing the latest proprietary models such as GPT-4o. Our code and dataset will be made publicly at https://videolucy.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑