基于上下文感知跨模态交互的密集视频字幕显式时空语义建模
Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction
AI总结：
针对现有密集视频字幕方法隐式建模无法捕捉时间连贯性与完整语义的问题，本文提出显式时空语义建模框架CACMI，通过两个核心模块融合时序与语义信息，在两个基准数据集上取得SOTA性能。
摘要：
密集视频字幕任务需同时对未修剪视频中的显著事件进行定位并生成对应字幕。现有方法主要依赖额外先验知识和先进多任务架构以获取有竞争力的性能，但这类流程采用基于帧级或碎片化视频特征的隐式建模，无法捕捉事件序列间的时间连贯性以及视觉上下文内的完整语义。为解决该问题，本文提出名为上下文感知跨模态交互（Context-Aware Cross-Modal Interaction, CACMI）的显式时空语义建模框架，可同时利用视频内部潜在时间特性与文本语料库的语言语义。该模型包含两个核心组件：跨模态帧聚合模块通过跨模态检索聚合相关帧，提取时间连贯、与事件对齐的文本特征；上下文感知特征增强模块利用查询引导注意力将视觉动态与伪事件语义相融合。在ActivityNet Captions和YouCook2数据集上的大量实验表明，CACMI在密集视频字幕任务上取得了当前最优性能。
英文摘要：
Dense video captioning jointly localizes and captions salient events in untrimmed videos. Recent methods primarily focus on leveraging additional prior knowledge and advanced multi-task architectures to achieve competitive performance. However, these pipelines rely on implicit modeling that uses frame-level or fragmented video features, failing to capture the temporal coherence across event sequences and comprehensive semantics within visual contexts. To address this, we propose an explicit temporal-semantic modeling framework called Context-Aware Cross-Modal Interaction (CACMI), which leverages both latent temporal characteristics within videos and linguistic semantics from text corpus. Specifically, our model consists of two core components: Cross-modal Frame Aggregation aggregates relevant frames to extract temporally coherent, event-aligned textual features through cross-modal retrieval; and Context-aware Feature Enhancement utilizes query-guided attention to integrate visual dynamics with pseudo-event semantics. Extensive experiments on the ActivityNet Captions and YouCook2 datasets demonstrate that CACMI achieves the state-of-the-art performance on dense video captioning task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
