WIRE:冻结视频语言模型的权重空间隐式表示编码
WIRE: Weight Space Implicit Representation Encoding of Frozen Video LLMs
AI总结:
WIRE 是一种通过将冻结视频语言模型的部分计算转移到权重低秩更新来降低长视频处理成本的方法,在 Qwen3-VL-8B 上可压缩视觉流、减少预填充时间并保持解码延迟,在 256 帧基准中准确率最优。
摘要:
视频语言模型将视频表示为解码器输入中的一长串视觉嵌入序列。每个视觉嵌入在预填充阶段都会被所有解码器层处理,存储在键值缓存中,并被每个生成的 token 关注,因此长视频会同时增加预填充、内存和生成成本。本文提出 WIRE(Weight Space Implicit Representation Encoding,权重空间隐式表示编码),通过将冻结视频语言模型中部分依赖视频的计算从输入激活转移到其权重的临时低秩更新,来降低这种重复成本。WIRE 保留了短的显式视觉上下文,因此该更新是对视频的修正而非替代。WIRE 首先使用问题条件双向选择性扫描将每一帧的全分辨率嵌入压缩为一小组载体 token,并物理收缩解码器输入,使每个解码器层、缓存条目和解码步骤都在更短的序列上运行。选定解码器层中的超网络会读取压缩后的前向传播,以及收缩过程中移除的全分辨率特征的读出值,并生成对注意力和前馈投影的 LoRA 更新。这些更新在预填充后合并,因此自回归解码运行未修改的冻结主干。在 Qwen3-VL-8B 上,每帧带有 16 个载体 token 的 WIRE 将视觉流压缩了 3.5 倍,在 256 帧时将解码器预填充时间减少了 3.35 倍,在 1024 帧时减少了 4.14 倍,按比例缩小了键值缓存,并使每个 token 的解码延迟大致保持恒定。在 256 帧的 9 个基准设置中,它在我们评估的高效视频语言模型中,在共同的 256 帧预算下获得了最高的平均准确率。
英文摘要:
Video language models represent a video as a long sequence of visual embeddings in a decoder's input. Every visual embedding is processed by every decoder layer during prefill, stored in the key value cache, and attended by every generated token, so a long video raises prefill, memory, and generation cost at once. We present WIRE (Weight Space Implicit Representation Encoding), which reduces this recurring cost by moving part of the video conditioned computation of a frozen video language model from its input activations into a temporary low rank update of its weights. WIRE keeps a short explicit visual context, so the update acts as a correction rather than a replacement of the video. WIRE first compresses the full resolution embeddings of each frame into a small set of carrier tokens using a question conditioned bidirectional selective scan, and physically contracts the decoder input so every decoder layer, cache entry, and decoding step operates on the shorter sequence. Hypernetworks at selected decoder layers read the compressed forward pass together with a readout of the full resolution features removed by contraction and generate LoRA updates to the attention and feedforward projections. The updates are merged after prefill, so autoregressive decoding runs the unmodified frozen backbone. On Qwen3-VL-8B, WIRE with 16 carrier tokens per frame contracts the visual stream by $3.5\times$, reduces decoder prefill time by $3.35\times$ at 256 frames and $4.14\times$ at 1024 frames, shrinks the key value cache proportionally, and keeps per token decoding latency approximately constant. Across nine benchmark settings at 256 frames, it obtains the highest average accuracy among efficient video language models we evaluate under a common 256 frame budget.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。