arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.16366cs.CVcs.AI

Fre-Res: 频率-残差视频令牌压缩用于高效的视频多模态大语言模型

Fre-Res: Frequency-Residual Video Token Compression for Efficient Video MLLMs

College of Computer Science, National University of Defense Technology(计算机科学学院,国防科技大学);Shien-Ming Wu School of Intelligent Engineering, South China University of Technology(智能工程谢民明伍学院,华南理工大学)

Yigui Feng, Qinglin Wang, Yang Liu, Jie Liu

AI总结:

Fre-Res通过分离空间和时间信息,实现视频令牌压缩,在保持细节精度的同时提升效率,适用于短时事件和长视频推理。

摘要:

视频多模态大语言模型面临空间保真度与时间覆盖度之间的矛盾:保留细粒度视觉细节需要大量空间令牌,而捕捉短暂事件需要密集的时间采样。我们提出Fre-Res,一种预算自适应的双轨视频令牌压缩框架,分别处理这两种证据形式。Fre-Res保留稀疏的高保真空间锚点,并通过紧凑的残差频域令牌表示密集的时间演变。具体而言,它对视觉潜在空间中的帧间残差轨迹应用时间1D-DCT,在其中观察到强低频集中。为对齐频域动态与原生视觉嵌入,Fre-Res引入了空间引导吸收器,将时间残差信息注入与空间锚点对应的令牌中。在细粒度短视频和长视频推理基准上,Fre-Res实现了有利的准确率-效率权衡,匹配或接近全令牌性能,同时显著减少视觉令牌长度。广泛消融实验进一步表明,时间频域残差保留因果转换线索,而空间锚点对细粒度物体和布局推理至关重要。

英文摘要:

Video MLLMs face a persistent tension between spatial fidelity and temporal coverage: preserving fine-grained visual details requires many spatial tokens, while capturing short-lived events requires dense temporal sampling. We propose \textbf{Fre-Res}, a budget-adaptive dual-track video-token compression framework that separates these two forms of evidence. Fre-Res preserves sparse high-fidelity spatial anchors and represents dense temporal evolution through compact residual-frequency tokens. Specifically, it applies temporal 1D-DCT to inter-frame residual trajectories in vision-latent space, where we observe strong low-frequency concentration. To align frequency-domain dynamics with native visual embeddings, Fre-Res introduces a Spatial-Guided Absorber that injects temporal residual information into spatially corresponding anchor tokens. Across fine-grained short-video and long-video reasoning benchmarks, Fre-Res achieves a favorable accuracy--efficiency trade-off, matching or approaching full-token performance while substantially reducing visual-token length. Extensive ablations further show that temporal-frequency residuals preserve causal transition cues, while spatial anchors remain essential for fine-grained object and layout reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑