SGCap：解码语义组用于零样本视频描述生成
SGCap: Decoding Semantic Group for Zero-shot Video Captioning
AI总结：
提出SGCap方法，通过语义组解码、关键句子选择和概率采样监督，利用多帧信息并建模时间动态，实现零样本视频描述生成，性能超越现有零样本方法并接近全监督方法。
摘要：
零样本视频描述生成旨在无需在视频-文本对上训练模型的情况下生成描述视频的句子，这一方向仍未被充分探索。现有的零样本图像描述生成方法通常采用纯文本训练范式，其中语言解码器重建从CLIP获得的单句嵌入。然而，直接将它们扩展到视频领域并非最优，因为对所有帧应用平均池化会忽略时间动态。为应对这一挑战，我们提出了一种用于零样本视频描述生成的语义组描述（SGCap）方法。具体而言，它开发了语义组解码（SGD）策略，以利用多帧信息并显式建模帧间时间关系。此外，现有的零样本描述生成方法依赖余弦相似度进行句子检索，并在单帧级描述的监督下重建描述，无法提供足够的视频级监督。为缓解这一问题，我们引入了两个关键组件，包括关键句子选择（KSS）模块和概率采样监督（PSS）模块。这两个模块构建了语义多样的句子组，这些句子组建模时间动态并引导模型捕获句子间的因果关系，从而增强其对视频描述生成的泛化能力。在多个基准上的实验结果表明，SGCap显著优于以往最先进的零样本替代方法，甚至达到了与全监督方法竞争的性能。代码可在https://github.com/mlvccn/SGCap_Video获取。
英文摘要：
Zero-shot video captioning aims to generate sentences for describing videos without training the model on video-text pairs, which remains underexplored. Existing zero-shot image captioning methods typically adopt a text-only training paradigm, where a language decoder reconstructs single-sentence embeddings obtained from CLIP. However, directly extending them to the video domain is suboptimal, as applying average pooling over all frames neglects temporal dynamics. To address this challenge, we propose a Semantic Group Captioning (SGCap) method for zero-shot video captioning. In particular, it develops the Semantic Group Decoding (SGD) strategy to employ multi-frame information while explicitly modeling inter-frame temporal relationships. Furthermore, existing zero-shot captioning methods that rely on cosine similarity for sentence retrieval and reconstruct the description supervised by a single frame-level caption, fail to provide sufficient video-level supervision. To alleviate this, we introduce two key components, including the Key Sentences Selection (KSS) module and the Probability Sampling Supervision (PSS) module. The two modules construct semantically-diverse sentence groups that models temporal dynamics and guide the model to capture inter-sentence causal relationships, thereby enhancing its generalization ability to video captioning. Experimental results on several benchmarks demonstrate that SGCap significantly outperforms previous state-of-the-art zero-shot alternatives and even achieves performance competitive with fully supervised ones. Code is available at https://github.com/mlvccn/SGCap_Video.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
