arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.04501cs.CV
收藏

从大语言模型学习视频表示

Learning Video Representations from Large Language Models

Meta AI;University of Texas, Austin(德克萨斯大学奥斯汀分校)

Yue Zhao, Ishan Misra, Philipp Krähenbühl, Rohit Girdhar

摘要:

我们介绍LaViLa,一种利用大语言模型(LLM)学习视频语言表示的新方法。我们重新利用预先训练的LLM以视觉输入为条件,并对其进行微调以创建自动视频叙述者。我们自动生成的叙述提供了许多优势,包括长视频的密集覆盖,视觉信息和文本的更好的时间同步,以及更高的文本多样性。与这些额外的自动生成的叙述相比,学习的视频-文本嵌入在多个第一人称和第三人称视频任务中的表现优于先前的最先进技术,无论是在zero-shot还是微调设置中。最值得注意的是,LaViLa在EGTEA分类上获得了10.1%的绝对增益,在Epic-Kitchen-100多实例检索基准测试中获得了5.9%的绝对增益。此外,LaViLa仅使用Ego 4D数据集一半的叙述进行训练,优于在完整集合上训练的基线模型,并且在增加预训练数据和模型大小时显示出积极的缩放行为。

英文摘要:

We introduce LaViLa, a new approach to learning video-language representations by leveraging Large Language Models (LLMs). We repurpose pre-trained LLMs to be conditioned on visual input, and finetune them to create automatic video narrators. Our auto-generated narrations offer a number of advantages, including dense coverage of long videos, better temporal synchronization of the visual information and text, and much higher diversity of text. The video-text embedding learned contrastively with these additional auto-generated narrations outperforms the previous state-of-the-art on multiple first-person and third-person video tasks, both in zero-shot and finetuned setups. Most notably, LaViLa obtains an absolute gain of 10.1% on EGTEA classification and 5.9% Epic-Kitchens-100 multi-instance retrieval benchmarks. Furthermore, LaViLa trained with only half the narrations from the Ego4D dataset outperforms baseline models trained on the full set, and shows positive scaling behavior on increasing pre-training data and model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑