通过联合优化文本与片段实现零样本密集视频字幕生成
Zero-Shot Dense Video Captioning by Jointly Optimizing Text and Moment
AI总结：
本文提出零样本密集视频字幕方法 ZeroTA，通过联合优化软时刻掩码与 GPT-2 前缀参数并结合 CLIP 和时间 IoU 损失，在无训练标注情况下定位并描述多个视频事件。
摘要：
密集视频字幕生成是一项定位视频中有意义的时刻并为其生成相关字幕的任务，通常需要庞大且昂贵的标注视频片段及对应文本语料。为尽量降低标注成本，我们提出 ZeroTA，这是一种以零样本方式进行密集视频字幕生成的新方法。该方法不需要任何视频或标注用于训练；相反，它在测试时仅针对输入进行优化，从而在每个输入视频中定位并描述事件。具体而言，我们引入表示视频中时间片段的软时刻掩码，并将其与语言模型的前缀参数联合优化。这种联合优化通过最大化生成文本与视频中某个时刻之间的匹配分数，使冻结的语言生成模型（即 GPT-2）与冻结的视觉-语言对比模型（即 CLIP）对齐。我们还引入成对时间 IoU 损失，使一组软时刻掩码能够捕获视频中的多个不同事件。该方法可有效发现视频中多样且重要的事件，生成的字幕也能恰当地描述这些事件。实验结果表明，在广泛使用的基准 ActivityNet Captions 上，ZeroTA 超过了零样本基线，甚至优于最先进的少样本方法。此外，在域外场景中评估时，与监督方法相比，我们的方法表现出更强的鲁棒性。本研究揭示了对齐语言生成模型和视觉-语言模型等常用模型的潜力，可解锁一种新能力：理解视频的时间方面。
英文摘要：
Dense video captioning, a task of localizing meaningful moments and generating relevant captions for videos, often requires a large, expensive corpus of annotated video segments paired with text. In an effort to minimize the annotation cost, we propose ZeroTA, a novel method for dense video captioning in a zero-shot manner. Our method does not require any videos or annotations for training; instead, it localizes and describes events within each input video at test time by optimizing solely on the input. This is accomplished by introducing a soft moment mask that represents a temporal segment in the video and jointly optimizing it with the prefix parameters of a language model. This joint optimization aligns a frozen language generation model (i.e., GPT-2) with a frozen vision-language contrastive model (i.e., CLIP) by maximizing the matching score between the generated text and a moment within the video. We also introduce a pairwise temporal IoU loss to let a set of soft moment masks capture multiple distinct events within the video. Our method effectively discovers diverse significant events within the video, with the resulting captions appropriately describing these events. The empirical results demonstrate that ZeroTA surpasses zero-shot baselines and even outperforms the state-of-the-art few-shot method on the widely-used benchmark ActivityNet Captions. Moreover, our method shows greater robustness compared to supervised methods when evaluated in out-of-domain scenarios. This research provides insight into the potential of aligning widely-used models, such as language generation models and vision-language models, to unlock a new capability: understanding temporal aspects of videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
