arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.02631cs.CV
收藏

对比语言视频时间预训练

Contrastive Language Video Time Pre-training

UC Riverside(加州大学河滨分校);Intel Labs(英特尔实验室)

Hengyue Liu, Kyle Min, Hector A. Valdez, Subarna Tripathi

AI总结:

本文提出 LAVITI 方法,通过可学习时刻查询和相对时间嵌入,在长视频中对比学习语言、视频与时间表征,并在 Ego4D 上以低计算成本训练,在 CharadesEgo 动作识别中取得最先进结果。

摘要:

我们提出 LAVITI,一种通过对比学习在长视频中学习语言、视频和时间表征的新方法。与 EgoVLP 等基于视频-文本对进行预训练的方法不同,LAVITI 旨在通过从未裁剪视频中提取有意义的时刻来对齐语言、视频和时间特征。我们的模型采用一组可学习的时刻查询来解码片段级的视觉、语言和时间特征。除了视觉与语言对齐之外,我们引入相对时间嵌入(TE)来表示视频中的时间戳,从而实现对时间的对比学习。与传统方法显著不同的是,对特定时间戳的预测通过计算预测 TE 与所有 TE 之间的相似度分数来完成。此外,现有的视频理解方法由于计算复杂度和内存占用较高,主要针对短视频设计。我们的方法可以在一天内仅使用 8 块 NVIDIA RTX-3090 GPU 在 Ego4D 数据集上完成训练。我们在 CharadesEgo 动作识别上验证了该方法,取得了最先进的结果。

英文摘要:

We introduce LAVITI, a novel approach to learning language, video, and temporal representations in long-form videos via contrastive learning. Different from pre-training on video-text pairs like EgoVLP, LAVITI aims to align language, video, and temporal features by extracting meaningful moments in untrimmed videos. Our model employs a set of learnable moment queries to decode clip-level visual, language, and temporal features. In addition to vision and language alignment, we introduce relative temporal embeddings (TE) to represent timestamps in videos, which enables contrastive learning of time. Significantly different from traditional approaches, the prediction of a particular timestamp is transformed by computing the similarity score between the predicted TE and all TEs. Furthermore, existing approaches for video understanding are mainly designed for short videos due to high computational complexity and memory footprint. Our method can be trained on the Ego4D dataset with only 8 NVIDIA RTX-3090 GPUs in a day. We validated our method on CharadesEgo action recognition, achieving state-of-the-art results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑