时间收益，空间代价：重新审视多模态大语言模型中的视频微调
Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models
AI总结：
本文研究了视频微调对多模态大语言模型视觉能力的影响，发现视频性能提升的同时，静态图像表现可能受限，提出混合帧策略以缓解空间与时间理解的平衡问题。
摘要：
多模态大语言模型（MLLMs）通常分阶段训练，视频监督微调（Video-SFT）是提升视觉理解的关键步骤。然而，其对视觉能力细粒度演变，特别是空间与时间理解的平衡影响尚不明确。本文系统研究了Video-SFT如何重塑MLLMs的视觉能力。在不同架构、参数规模和帧采样设置下，观察到一致模式：Video-SFT可靠提升视频性能，但常导致静态图像基准有限增益或退化。进一步表明，这种权衡与时间预算密切相关：增加采样帧数通常提升视频性能，但不保证提升静态图像性能。受此发现启发，我们研究了指令感知的混合帧策略，适配分配帧数并部分缓解图像-视频权衡。结果表明，Video-SFT并非多模态大语言模型的免费午餐，保持空间理解仍是联合图像-视频训练的核心挑战。
英文摘要：
Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
