LongVU-TTT：用于长视频理解中视觉重采样的因果测试时训练方法
LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding
浏览论文内容
中文总结 AI 辅助
LongVU-TTT在视觉编码器与LLM间插入带因果快速权重更新的卷积TTT重采样器，通过混合选择器保留帧证据，在五个视频理解基准测试中性能具竞争力，较多种基线方法有明显提升。
中文摘要 AI 辅助
长视频多模态大语言模型（MLLMs）必须对时间变化进行建模，而有限的视觉令牌预算会消除大部分帧证据。本文提出LongVU-TTT，它在视觉编码器与大语言模型（LLM）之间插入了一个带有因果快速权重更新的卷积测试时训练（TTT）重采样器。其分组2D快速权重可适配每个视频，在压缩前对帧特征进行上下文化处理，同时混合均匀与变化感知选择器会为下游推理保留显式视觉证据。在受控条件下，TTT-Conv在MLVU数据集上较TTT-MLP提升最高达+2.12，较双向Mamba2提升最高达+3.04，且在三个基准测试中均优于注意力型和固定状态循环重采样器。分析表明，快速权重表现为时间聚合状态而非可靠的长程情景记忆：其益处会随证据变远而衰减，这促使需显式保留帧。LongVU-TTT最多可处理512帧，再将其缩减为128个LLM帧，并在五个视频理解基准测试中取得了有竞争力的性能。
英文摘要
Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.
发表机构
- KAUST（阿卜杜拉国王科技大学）
- Snowflake（雪花公司）
- Microsoft Inc.（微软公司）
- Jülich Supercomputing Centre, Forschungszentrum Jülich（于利希研究中心于利希超级计算中心）
机构由 AI 辅助整理，请以论文原文为准。
