Video2LoRA: 视觉-语言模型的参数化视频内化
Frames2LoRA: Parametric Video Internalization for Vision-Language Models
AI总结：
提出Video2LoRA方法，通过感知器超网络从视频编码中直接生成LoRA适配器，实现零视觉令牌的视频查询，在保持性能的同时大幅降低计算成本。
摘要：
在视觉-语言模型中处理视频成本高昂：每帧占用数百个令牌，推理成本随每帧和每次重复查询而增加。我们引入Video2LoRA，一种参数化视频内化方法。感知器超网络逐层读取冻结VLM编码视频时产生的中间表示，并在单次前向传播中生成低秩适配（LoRA）适配器。与需要迭代梯度更新的标准LoRA微调不同，Video2LoRA直接从视频预测这些权重。在SmolVLM2 500M和2.2B上针对视频摘要和描述进行训练后，Video2LoRA使得相同的冻结VLM能够仅通过适配器回答查询，在查询时上下文中零视觉令牌。Video2LoRA在两种模型规模的所有五个描述基准测试中，以及在八个视频问答基准测试-规模配对中的七个上，统计上非劣效且等同于直接视频上下文推理。尽管仅在12帧384px上训练，它在高达1024帧和1024px时仍保持稳定，而直接视频上下文推理通常会退化。在此扫描中，它将回答时的视觉令牌负载减少高达1500倍，查询TTFT减少6-80倍，同时保持视频忠实输出。我们还发现，为非重叠视频段独立生成的适配器可以在秩空间中组合，这为分块长视频内化提供了一条路径。
英文摘要：
Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Frames2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Frames2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Frames2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Frames2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
