arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.06708cs.CVcs.AIcs.CL
收藏

动词在行动中:提升视频-语言模型中的动词理解

Verbs in Action: Improving verb understanding in video-language models

University of Oxford(牛津大学);Google Research(谷歌研究院)

Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, Cordelia Schmid

AI总结:

本文提出动词聚焦对比(VFC)框架,通过利用大语言模型生成困难负样本并施加细粒度动词短语对齐损失,显著提升了基于CLIP的视频-语言模型在动词理解相关任务上的零样本性能。

摘要:

理解动词对于建模人与物体如何通过空间和时间彼此及与环境交互至关重要。最近,基于CLIP的最先进视频-语言模型已被证明动词理解能力有限,并且严重依赖名词,这限制了它们在需要动作和时间理解的现实世界视频应用中的表现。在这项工作中,我们通过提出一种新的动词聚焦对比(VFC)框架来提升基于CLIP的视频-语言模型的动词理解。该框架包含两个主要组成部分:(1)利用预训练大语言模型(LLMs)为跨模态对比学习创建困难负样本,同时采用一种校准策略来平衡正负样本对中概念的出现;(2)强制实施一种细粒度的动词短语对齐损失。我们的方法在三个专注于动词理解的下游任务上取得了零样本性能的最先进结果:视频-文本匹配、视频问答和视频分类。据我们所知,这是首个提出缓解动词理解问题的方法的工作,而不仅仅是强调该问题。

英文摘要:

Understanding verbs is crucial to modelling how people and objects interact with each other and the environment through space and time. Recently, state-of-the-art video-language models based on CLIP have been shown to have limited verb understanding and to rely extensively on nouns, restricting their performance in real-world video applications that require action and temporal understanding. In this work, we improve verb understanding for CLIP-based video-language models by proposing a new Verb-Focused Contrastive (VFC) framework. This consists of two main components: (1) leveraging pretrained large language models (LLMs) to create hard negatives for cross-modal contrastive learning, together with a calibration strategy to balance the occurrence of concepts in positive and negative pairs; and (2) enforcing a fine-grained, verb phrase alignment loss. Our method achieves state-of-the-art results for zero-shot performance on three downstream tasks that focus on verb understanding: video-text matching, video question-answering and video classification. To the best of our knowledge, this is the first work which proposes a method to alleviate the verb understanding problem, and does not simply highlight it.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑