预训练图文模型其实是隐藏的视频字幕生成器
Pretrained Image-Text Models are Secretly Video Captioners
AI总结：
本文提出仅用6,000个视频-文本对后训练 BLIP2 并简单拼接帧，将图像字幕模型转化为高性能视频字幕器，在多个基准上达到顶级排名。
摘要：
开发视频字幕模型在计算上代价高昂。视频的动态特性也使多模态模型的设计更加复杂，而这些模型需要有效地为这些序列生成字幕。然而，我们发现，只需使用极少的计算资源，并且无需为处理视频动态进行复杂修改，一个基于图像的模型就可以被重新利用，其表现优于若干专门的视频字幕系统。我们适配后的模型在主要基准测试上展现出顶级性能，在 MSRVTT 和 MSVD 上排名第2，在 VATEX 上排名第3。我们仅使用6,000个视频-文本对，并通过简单拼接帧的方式，对典型图像字幕模型 BLIP2 进行后训练，将其转化为具有竞争力的视频字幕生成器；这一数据量远少于其他方法所用的250万至1.44亿个对。从资源优化角度看，本视频字幕研究聚焦三个基本因素：优化模型规模、最大化数据效率，以及引入强化学习。这项广泛研究表明，轻量级、基于图像的适配策略能够匹敌最先进的视频字幕系统，为低资源场景提供了实用解决方案。
英文摘要：
Developing video captioning models is computationally expensive. The dynamic nature of video also complicates the design of multimodal models that can effectively caption these sequences. However, we find that by using minimal computational resources and without complex modifications to address video dynamics, an image-based model can be repurposed to outperform several specialised video captioning systems. Our adapted model demonstrates top tier performance on major benchmarks, ranking 2nd on MSRVTT and MSVD, and 3rd on VATEX. We transform it into a competitive video captioner by post training a typical image captioning model BLIP2 with only 6,000 video text pairs and simply concatenating frames (significantly fewer data than other methods), which use 2.5 to 144 million pairs. From a resource optimization perspective, this video captioning study focuses on three fundamental factors: optimizing model scale, maximizing data efficiency, and incorporating reinforcement learning. This extensive study demonstrates that a lightweight, image based adaptation strategy can rival state-of-the-art video captioning systems, offering a practical solution for low-resource scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
