arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.10434cs.CVcs.MM
收藏

视频片段检索的逐帧跨模态匹配

Frame-wise Cross-modal Matching for Video Moment Retrieval

Xi’an Jiaotong University(西安交通大学);Shandong Artificial Intelligence Institute(山东人工智能研究院);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));Shandong Jianzhu University(山东建筑大学)

Haoyu Tang, Jihua Zhu, Meng Liu, Zan Gao, Zhiyong Cheng

AI总结:

本文提出 ACRM 模型,通过交互建模预测时间边界,引入注意力机制为语义丰富的查询词分配更高权重,并利用内部帧额外预测器提升定位精度,在 TACoS 和 Charades-STA 数据集上优于现有方法。

摘要:

视频片段检索旨在针对给定的语言查询,在视频中检索出一个片段。该任务的挑战包括:1) 需要在未剪辑视频中定位相关片段;2) 弥合文本查询与视频内容之间的语义鸿沟。为解决这些问题,早期方法采用滑动窗口或均匀采样先收集视频片段,再将每个片段与查询匹配。显然,这些策略耗时且常因黄金片段长度不可预测而导致定位精度不佳。为避免这些限制,研究者最近尝试直接预测相关片段边界,无需先生成视频片段。一种主流方法是为目标查询和视频帧生成多模态特征向量(如拼接),然后在该向量上使用回归方法进行边界检测。尽管此法取得一定进展,但我们认为这些方法未能很好地捕捉查询与视频帧之间的跨模态交互。本文提出了一种 Attentive Cross-modal Relevance Matching (ACRM) 模型,基于交互建模预测时间边界。此外,引入注意力模块为语义线索更丰富的查询词分配更高权重,这些词被认为对查找相关视频内容更重要。另一贡献是提出一个额外预测器,利用模型训练中的内部帧来提升定位精度。在 TACoS 和 Charades-STA 两个数据集上的大量实验表明,我们的方法优于多种 state-of-the-art 方法。还进行了消融实验以检验 ACRM 模型中不同模块的有效性。

英文摘要:

Video moment retrieval targets at retrieving a moment in a video for a given language query. The challenges of this task include 1) the requirement of localizing the relevant moment in an untrimmed video, and 2) bridging the semantic gap between textual query and video contents. To tackle those problems, early approaches adopt the sliding window or uniform sampling to collect video clips first and then match each clip with the query. Obviously, these strategies are time-consuming and often lead to unsatisfied accuracy in localization due to the unpredictable length of the golden moment. To avoid the limitations, researchers recently attempt to directly predict the relevant moment boundaries without the requirement to generate video clips first. One mainstream approach is to generate a multimodal feature vector for the target query and video frames (e.g., concatenation) and then use a regression approach upon the multimodal feature vector for boundary detection. Although some progress has been achieved by this approach, we argue that those methods have not well captured the cross-modal interactions between the query and video frames. In this paper, we propose an Attentive Cross-modal Relevance Matching (ACRM) model which predicts the temporal boundaries based on an interaction modeling. In addition, an attention module is introduced to assign higher weights to query words with richer semantic cues, which are considered to be more important for finding relevant video contents. Another contribution is that we propose an additional predictor to utilize the internal frames in the model training to improve the localization accuracy. Extensive experiments on two datasets TACoS and Charades-STA demonstrate the superiority of our method over several state-of-the-art methods. Ablation studies have been also conducted to examine the effectiveness of different modules in our ACRM model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑