魔鬼在分布中：显式建模场景内容是零样本视频字幕生成的关键
The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning
AI总结：
针对零样本视频字幕生成中现有方法忽略部分视觉输入的问题，提出渐进式多粒度文本提示策略与类别感知检索机制，在多个基准上显著提升CIDEr指标。
摘要：
零样本视频字幕生成要求模型在没有人工标注的视频-文本对进行训练的情况下生成高质量字幕。该问题的最先进方法利用CLIP提取视觉相关的文本提示，以指导语言模型生成字幕。这些方法往往只关注场景的一个关键方面，构建字幕时忽略了其余视觉输入。为了解决这一问题并生成更准确、完整的字幕，我们提出了一种用于零样本视频字幕生成的渐进式多粒度文本提示策略。我们的方法构建了三个不同的记忆库，包含名词短语、名词短语的场景图以及完整句子。此外，我们引入了一种类别感知检索机制，对所讨论特定主题周围的自然语言分布进行建模。大量实验证明了我们方法的有效性，与现有最先进方法相比，在MSR-VTT、MSVD和VATEX基准上的主要指标CIDEr分别提升了5.7%、16.2%和3.4%。
英文摘要：
Zero-shot video captioning requires that a model generate high-quality captions without human-annotated video-text pairs for training. State-of-the-art approaches to the problem leverage CLIP to extract visual-relevant textual prompts to guide language models in generating captions. These methods tend to focus on one key aspect of the scene and build a caption that ignores the rest of the visual input. To address this issue, and generate more accurate and complete captions, we propose a novel progressive multi-granularity textual prompting strategy for zero-shot video captioning. Our approach constructs three distinct memory banks, encompassing noun phrases, scene graphs of noun phrases, and entire sentences. Moreover, we introduce a category-aware retrieval mechanism that models the distribution of natural language surrounding the specific topics in question. Extensive experiments demonstrate the effectiveness of our method with 5.7%, 16.2%, and 3.4% improvements in terms of the main metric CIDEr on MSR-VTT, MSVD, and VATEX benchmarks compared to existing state-of-the-art.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
