SAIL：基于相似性感知引导和跨字幕增强学习的弱监督密集视频字幕生成
SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning
AI总结：
SAIL通过跨模态对齐和大语言模型增强策略，提升弱监督密集视频字幕生成的准确性和语义感知能力。
摘要：
弱监督密集视频字幕生成旨在仅利用字幕标注训练，无需时间边界即可定位并描述视频中的事件。先前工作引入了基于高斯遮罩和互补字幕的隐式监督范式。然而，现有方法仅关注生成不重叠的遮罩，而未考虑其与对应事件的语义关系，导致生成的遮罩简单且分布均匀，无法捕捉具有语义意义的区域。此外，仅依赖真实字幕会导致性能欠佳，因为现有数据集本身存在固有的稀疏性。在本工作中，我们提出SAIL，通过跨模态对齐构建语义感知的遮罩。我们的相似性感知训练目标引导遮罩强调与对应事件字幕相似度高的视频区域。此外，为了在稀疏标注设置下引导更准确的遮罩生成，我们引入基于大语言模型的增强策略，生成合成字幕以提供额外的对齐信号。这些合成字幕通过跨遮罩机制进行整合，为精确的时间定位提供辅助指导，而不会影响主要目标。在ActivityNet Captions和YouCook2数据集上的实验表明，SAIL在字幕生成和定位指标上均达到最先进的性能。
英文摘要：
Weakly-Supervised Dense Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries. Prior work introduced an implicit supervision paradigm based on Gaussian masking and complementary captioning. However, existing method focuses merely on generating non-overlapping masks without considering their semantic relationship to corresponding events, resulting in simplistic, uniformly distributed masks that fail to capture semantically meaningful regions. Moreover, relying solely on ground-truth captions leads to sub-optimal performance due to the inherent sparsity of existing datasets. In this work, we propose SAIL, which constructs semantically-aware masks through cross-modal alignment. Our similarity aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions. Furthermore, to guide more accurate mask generation under sparse annotation settings, we introduce an LLM-based augmentation strategy that generates synthetic captions to provide additional alignment signals. These synthetic captions are incorporated through an inter-mask mechanism, providing auxiliary guidance for precise temporal localization without degrading the main objective. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance on both captioning and localization metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
