VLANet：面向弱监督视频片段检索的视频-语言对齐网络
VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval
AI总结：
针对弱监督视频片段检索中候选片段多、查询表示粗及单向注意力导致定位性能受限的问题，提出VLANet，通过代理候选选择和级联跨模态注意力学习清晰对齐，在两个数据集上达SOTA。
摘要：
视频片段检索（Video Moment Retrieval, VMR）是从未修剪视频中定位自然语言查询所指定时间片段的任务。针对VMR，已有多种需要全监督训练的方法被提出。但获取大量带有每个查询对应时间边界标注的训练视频是一项劳动密集型工作。本文探索弱监督下的VMR方法（wVMR）：训练时无需时间片段标注，仅使用描述视频片段的文本查询。现有的wVMR方法生成多尺度候选片段，并应用查询引导的注意力机制突出最相关的候选片段。为利用弱监督信号，这类方法采用对比学习，使正确的视频-查询对得分高于错误对。研究发现，大量候选片段、粗粒度的查询表示以及单向注意力机制会导致注意力图模糊，限制了定位性能。为解决该问题，本文提出视频-语言对齐网络（Video-Language Alignment Network, VLANet），通过剔除虚假候选片段、结合细粒度查询表示的多向注意力机制，学习更清晰的注意力。其中，代理候选片段选择模块基于联合嵌入空间中与查询的邻近度选择候选片段，大幅减少候选数量，既降低计算负载又使注意力更集中。随后，级联跨模态注意力模块考虑密集特征交互和多向注意力流，学习多模态对齐。VLANet采用对比损失端到端训练，促使语义相似的视频和查询相互靠近。实验表明，该方法在Charades-STA和DiDeMo数据集上取得了当前最优性能。
英文摘要：
Video Moment Retrieval (VMR) is a task to localize the temporal moment in untrimmed video specified by natural language query. For VMR, several methods that require full supervision for training have been proposed. Unfortunately, acquiring a large number of training videos with labeled temporal boundaries for each query is a labor-intensive process. This paper explores methods for performing VMR in a weakly-supervised manner (wVMR): training is performed without temporal moment labels but only with the text query that describes a segment of the video. Existing methods on wVMR generate multi-scale proposals and apply query-guided attention mechanisms to highlight the most relevant proposal. To leverage the weak supervision, contrastive learning is used which predicts higher scores for the correct video-query pairs than for the incorrect pairs. It has been observed that a large number of candidate proposals, coarse query representation, and one-way attention mechanism lead to blurry attention maps which limit the localization performance. To handle this issue, Video-Language Alignment Network (VLANet) is proposed that learns sharper attention by pruning out spurious candidate proposals and applying a multi-directional attention mechanism with fine-grained query representation. The Surrogate Proposal Selection module selects a proposal based on the proximity to the query in the joint embedding space, and thus substantially reduces candidate proposals which leads to lower computation load and sharper attention. Next, the Cascaded Cross-modal Attention module considers dense feature interactions and multi-directional attention flow to learn the multi-modal alignment. VLANet is trained end-to-end using contrastive loss which enforces semantically similar videos and queries to gather. The experiments show that the method achieves state-of-the-art performance on Charades-STA and DiDeMo datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
