arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.19554cs.CV
收藏

通过高效后预训练收获视频基础模型

Harvest Video Foundation Models via Efficient Post-Pretraining

Yizhuo Li, Kunchang Li, Yinan He, Yi Wang, Yali Wang, Limin Wang, Yu Qiao, Ping Luo

AI总结:

本文提出一种高效框架,通过随机丢弃视频片段和遮蔽文本来从图像基础模型中获取视频基础模型,提升了训练效率并强化了跨模态融合,实验表明其在多种视频-语言任务中达到SOTA性能。

摘要:

构建视频-语言基础模型因视频数据冗余性和高质量数据集缺乏而成本高昂且困难。本文提出一种高效框架,通过在后预训练过程中随机丢弃视频片段和遮蔽文本来获取视频基础模型。我们的方法通过随机丢弃视频片段显著提升了训练效率,同时通过遮蔽文本强化了跨模态融合。我们进行了广泛的实验,验证了该方法在多种视频-语言下游任务中的有效性,包括各种零样本任务、视频问答和视频-文本检索。尽管方法简单,但其性能可与一些高度预训练的视频基础模型相媲美。我们的方法极其高效,可在8块GPU上不到一天内完成训练,仅需WebVid-10M作为预训练数据。我们希望该方法能成为流行视频基础模型的简单而强大的替代品,为构建这些模型提供有用的见解,并使大规模预训练模型更加可访问和可持续。这是InternVideo项目的一部分(https://github.com/OpenGVLab/InternVideo)

英文摘要:

Building video-language foundation models is costly and difficult due to the redundant nature of video data and the lack of high-quality video-language datasets. In this paper, we propose an efficient framework to harvest video foundation models from image ones. Our method is intuitively simple by randomly dropping input video patches and masking out input text during the post-pretraining procedure. The patch dropping boosts the training efficiency significantly and text masking enforces the learning of cross-modal fusion. We conduct extensive experiments to validate the effectiveness of our method on a wide range of video-language downstream tasks including various zero-shot tasks, video question answering, and video-text retrieval. Despite its simplicity, our method achieves state-of-the-art performances, which are comparable to some heavily pretrained video foundation models. Our method is extremely efficient and can be trained in less than one day on 8 GPUs, requiring only WebVid-10M as pretraining data. We hope our method can serve as a simple yet strong counterpart for prevalent video foundation models, provide useful insights when building them, and make large pretrained models more accessible and sustainable. This is part of the InternVideo project https://github.com/OpenGVLab/InternVideo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑