范式转变：面向视频中的时序句子定位的端到端训练
A Paradigm Shift: Fully End-to-End Training for Temporal Sentence Grounding in Videos
AI总结：
本文提出一种端到端训练范式，联合优化视频主干网络和定位头，通过引入Sentence Conditioned Adapter提升视觉表征，实验表明优于现有方法。
摘要：
视频中的时序句子定位（TSGV）旨在从未剪辑视频中定位与句子查询语义对应的时序片段。大多数现有方法采用预训练的查询无关视觉编码器进行离线特征提取，且视频主干网络被冻结且未针对TSGV优化，导致任务不匹配问题。为弥合这一差距，我们提出了一种完全端到端范式，联合优化视频主干网络和定位头。我们首先进行了实证研究，验证了端到端学习在不同模型规模下优于冻结基线的有效性。此外，我们引入了Sentence Conditioned Adapter（SCADA），利用句子特征来训练视频主干网络的少量参数。SCADA通过精确整合语言嵌入来调节特征图，从而在减少内存消耗的同时部署更深层次的网络主干，并显著提升视觉表征。在两个基准测试中，我们的方法优于现有最先进方法。代码和模型将被发布。
英文摘要：
Temporal sentence grounding in videos (TSGV) aims to localize a temporal segment that semantically corresponds to a sentence query from an untrimmed video. Most current methods adopt pre-trained query-agnostic visual encoders for offline feature extraction, and the video backbones are frozen and not optimized for TSGV. This leads to a task discrepancy issue for the video backbone trained for visual classification, but utilized for TSGV. To bridge this gap, we propose a fully end-to-end paradigm that jointly optimizes the video backbone and localization head. We first conduct an empirical study validating the effectiveness of end-to-end learning over frozen baselines across different model scales. Furthermore, we introduce a Sentence Conditioned Adapter (SCADA), which leverages sentence features to train a small portion of video backbone parameters adaptively. SCADA facilitates the deployment of deeper network backbones with reduced memory and significantly enhances visual representation by modulating feature maps through precise integration of linguistic embeddings. Experiments on two benchmarks show that our method outperforms state-of-the-art approaches. The code and models will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
