arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.07046cs.CV
收藏

RETTA:面向零样本视频字幕生成的检索增强测试时自适应方法

RETTA: Retrieval-Enhanced Test-Time Adaptation for Zero-Shot Video Captioning

University of Chinese Academy of Science, Beijing,100190, China(中国科学院大学);Macquarie University(麦考瑞大学)

Yunchuan Ma, Laiyun Qing, Guorong Li, Yuankai Qi, Amin Beheshti, Quan Z. Sheng, Qingming Huang

AI总结:

针对零样本视频字幕生成研究不足的问题,提出RETTA框架,以可学习token为媒介连接四个冻结预训练模型,通过推理数据软标签在测试时自适应学习,在三个数据集上CIDEr指标较现有SOTA提升5.1%-32.4%。

摘要:

尽管全监督视频字幕生成已取得显著进展,但零样本方法的相关研究仍较为匮乏。本文提出一种名为Retrieval-Enhanced Test-Time Adaptation(检索增强测试时自适应,简称RETTA)的新型零样本视频字幕生成框架,该框架利用现有预训练的大规模视觉-语言模型,通过测试时自适应直接生成字幕。具体而言,我们借助四个核心模型搭建视频与文本的桥梁:通用视频-文本检索模型XCLIP、通用图像-文本匹配模型CLIP、文本对齐模型AnglE以及文本生成模型GPT-2,选择这些模型是因为其源代码公开可用。该研究的核心挑战在于如何让文本生成模型充分感知给定视频的内容,从而生成对应的字幕。为解决这一问题,我们提出使用可学习token作为GPT-2、XCLIP、CLIP和AnglE这四个冻结模型之间的通信媒介。与传统使用训练数据训练这些token的方式不同,我们提出在若干精心设计的损失函数下,利用推理数据的软标签学习这些token,使token能够吸收适配GPT-2的视频信息。该过程仅需少量迭代即可高效完成(实验中使用16次迭代),且无需真值数据。在MSR-VTT、MSVD和VATEX三个广泛使用的数据集上的大量实验结果表明,相较于多种当前最优的零样本视频字幕生成方法,本方法在核心指标CIDEr上取得了5.1%-32.4%的绝对提升。

英文摘要:

Despite the significant progress of fully-supervised video captioning, zero-shot methods remain much less explored. In this paper, we propose a novel zero-shot video captioning framework named Retrieval-Enhanced Test-Time Adaptation (RETTA), which takes advantage of existing pretrained large-scale vision and language models to directly generate captions with test-time adaptation. Specifically, we bridge video and text using four key models: a general video-text retrieval model XCLIP, a general image-text matching model CLIP, a text alignment model AnglE, and a text generation model GPT-2, due to their source-code availability. The main challenge is how to enable the text generation model to be sufficiently aware of the content in a given video so as to generate corresponding captions. To address this problem, we propose using learnable tokens as a communication medium among these four frozen models GPT-2, XCLIP, CLIP, and AnglE. Different from the conventional way that trains these tokens with training data, we propose to learn these tokens with soft targets of the inference data under several carefully crafted loss functions, which enable the tokens to absorb video information catered for GPT-2. This procedure can be efficiently done in just a few iterations (we use 16 iterations in the experiments) and does not require ground truth data. Extensive experimental results on three widely used datasets, MSR-VTT, MSVD, and VATEX, show absolute 5.1%-32.4% improvements in terms of the main metric CIDEr compared to several state-of-the-art zero-shot video captioning methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑