arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.06129cs.CV
收藏

在数百万视频上蒸馏视觉-语言模型

Distilling Vision-Language Models on Millions of Videos

Google Research(谷歌研究院);University of Texas, Austin(德克萨斯大学奥斯汀分校)

Yue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu, Chun-Te Chu, Hui Miao, Florian Schroff, Hartwig Adam, Ting Liu, Boqing Gong, Philipp Krähenbühl, Liangzhe Yuan

AI总结:

针对视频-文本数据不足,本文用合成教学数据从图像-语言基线微调视频模型,并以 VIIT 自动标注数百万视频蒸馏双编码器,在多项视频问答与检索基准上取得显著提升。

摘要:

视觉-语言模型近来的进展在很大程度上归功于丰富的图文数据。我们希望将这一成功复制到视频-语言模型上,但现实中根本没有足够的人工策划视频-文本数据。因此,我们借助合成的教学数据,从强大的图像-语言基线出发微调一个视频-语言模型。随后,通过视频教学微调(VIIT)得到的视频模型被用于自动标注数百万个视频,以生成高质量字幕。我们表明,经过适配的视频-语言模型在广泛的视频-语言基准上表现良好。例如,它在开放式 NExT-QA 上比此前最佳结果高出 2.8%。此外,我们的模型能够为此前未见过的视频生成详细描述,这些描述比现有方法提供了更好的文本监督。实验表明,在这些自动生成字幕上进行对比训练的视频-语言双编码器模型,比同样利用视觉-语言模型的最强基线好 3.8%。我们的最佳模型在 MSR-VTT 零样本文本到视频检索上优于最先进方法 6%。作为副产品,我们生成了迄今为止最大的视频字幕数据集。

英文摘要:

The recent advance in vision-language models is largely attributed to the abundance of image-text data. We aim to replicate this success for video-language models, but there simply is not enough human-curated video-text data available. We thus resort to fine-tuning a video-language model from a strong image-language baseline with synthesized instructional data. The resulting video model by video-instruction-tuning (VIIT) is then used to auto-label millions of videos to generate high-quality captions. We show the adapted video-language model performs well on a wide range of video-language benchmarks. For instance, it surpasses the best prior result on open-ended NExT-QA by 2.8%. Besides, our model generates detailed descriptions for previously unseen videos, which provide better textual supervision than existing methods. Experiments show that a video-language dual-encoder model contrastively trained on these auto-generated captions is 3.8% better than the strongest baseline that also leverages vision-language models. Our best model outperforms state-of-the-art methods on MSR-VTT zero-shot text-to-video retrieval by 6%. As a side product, we generate the largest video caption dataset to date.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑