探索时间事件线索用于密集视频描述中的循环共学习
Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-learning
AI总结：
本文提出多概念循环学习（MCCL）网络，通过帧级概念检测提供时间事件线索，并设计生成器与定位器间的循环共学习，在ActivityNet Captions和YouCook2上取得最先进性能。
摘要：
密集视频描述旨在检测和描述未修剪视频中的所有事件。本文提出了一种名为多概念循环学习（MCCL）的密集视频描述网络，其目标是：（1）在帧级别检测多个概念，利用这些概念增强视频特征并提供时间事件线索；（2）在描述网络内的生成器与定位器之间设计循环共学习，以促进语义感知和事件定位。具体而言，我们对每一帧执行弱监督概念检测，并将检测到的概念嵌入集成到视频特征中以提供事件线索。此外，引入了视频级别的概念对比学习以获得更具判别性的概念嵌入。在描述网络中，我们建立了一种循环共学习策略，其中生成器通过语义匹配指导定位器进行事件定位，而定位器通过位置匹配增强生成器的事件语义感知，使语义感知和事件定位相互促进。MCCL在ActivityNet Captions和YouCook2数据集上达到了最先进的性能。大量实验证明了其有效性和可解释性。
英文摘要：
Dense video captioning aims to detect and describe all events in untrimmed videos. This paper presents a dense video captioning network called Multi-Concept Cyclic Learning (MCCL), which aims to: (1) detect multiple concepts at the frame level, using these concepts to enhance video features and provide temporal event cues; and (2) design cyclic co-learning between the generator and the localizer within the captioning network to promote semantic perception and event localization. Specifically, we perform weakly supervised concept detection for each frame, and the detected concept embeddings are integrated into the video features to provide event cues. Additionally, video-level concept contrastive learning is introduced to obtain more discriminative concept embeddings. In the captioning network, we establish a cyclic co-learning strategy where the generator guides the localizer for event localization through semantic matching, while the localizer enhances the generator's event semantic perception through location matching, making semantic perception and event localization mutually beneficial. MCCL achieves state-of-the-art performance on the ActivityNet Captions and YouCook2 datasets. Extensive experiments demonstrate its effectiveness and interpretability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
