FreshMem: 基于大脑的频率-空间混合内存用于流视频理解
FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding
AI总结：
FreshMem通过频率-空间混合内存网络，提升流视频理解的连续感知能力，实现短期保真与长期一致性的平衡。
摘要：
将多模态大语言模型（MLLMs）从离线模式过渡到在线流视频理解对于连续感知至关重要。然而，现有方法缺乏灵活的适应性，导致不可逆的细节丢失和上下文碎片化。为此，我们提出了FreshMem，一种受大脑对数感知和记忆巩固启发的频率-空间混合内存网络。FreshMem通过两个协同模块协调短期保真度与长期一致性：多尺度频率内存（MFM），将溢出帧投影到代表性的频率系数，并通过残差细节重建全局历史“ gist”；以及空间缩略图内存（STM），通过采用自适应压缩策略将连续流离散化为事件簇，将其压缩成高密度空间缩略图。大量实验表明，FreshMem显著提升了Qwen2-VL基线，分别在StreamingBench、OV-Bench和OVO-Bench上获得5.20%、4.52%和2.34%的提升。作为无需训练的解决方案，FreshMem优于几种完全微调的方法，提供了一种高效的长时域流视频理解范式。
英文摘要：
Transitioning Multimodal Large Language Models (MLLMs) from offline to online streaming video understanding is essential for continuous perception. However, existing methods lack flexible adaptivity, leading to irreversible detail loss and context fragmentation. To resolve this, we propose FreshMem, a Frequency-Space Hybrid Memory network inspired by the brain's logarithmic perception and memory consolidation. FreshMem reconciles short-term fidelity with long-term coherence through two synergistic modules: Multi-scale Frequency Memory (MFM), which projects overflowing frames into representative frequency coefficients, complemented by residual details to reconstruct a global historical "gist"; and Space Thumbnail Memory (STM), which discretizes the continuous stream into episodic clusters by employing an adaptive compression strategy to distill them into high-density space thumbnails. Extensive experiments show that FreshMem significantly boosts the Qwen2-VL baseline, yielding gains of 5.20%, 4.52%, and 2.34% on StreamingBench, OV-Bench, and OVO-Bench, respectively. As a training-free solution, FreshMem outperforms several fully fine-tuned methods, offering a highly efficient paradigm for long-horizon streaming video understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
