arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1912.03590cs.CVcs.IRcs.MM
收藏

基于自然语言的二维时间邻接网络矩定位算法

Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language

University of Rochester(罗切斯特大学);Microsoft Research(微软研究院)

Songyang Zhang, Houwen Peng, Jianlong Fu, Jiebo Luo

摘要:

我们解决的问题,检索一个特定的时刻,从一个未修剪的视频查询语句。这是一个具有挑战性的问题,因为目标时刻可能与未修剪视频中的其他时间时刻相关地发生。现有的方法不能很好地解决这一挑战,因为他们考虑时间的时刻单独和忽视的时间依赖性。在本文中,我们通过一个二维映射来建模视频时刻之间的时间关系,其中一个维度表示时刻的开始时间,另一个维度表示时刻的结束时间。该二维时间图可以覆盖不同长度的视频时刻,同时表示它们的相邻关系。基于2D地图,我们提出了时间邻近网络(2D-TAN),这是一种用于时刻定位的单镜头框架。它能够编码相邻的时间关系,同时学习判别特征,用于匹配视频时刻与参考表达式。我们在三个具有挑战性的基准上评估所提出的2D-TAN,即,Charades-STA、ActivityNet Captions和TACoS,我们的2D-TAN性能优于最先进的技术。

英文摘要:

We address the problem of retrieving a specific moment from an untrimmed video by a query sentence. This is a challenging problem because a target moment may take place in relations to other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they consider temporal moments individually and neglect the temporal dependencies. In this paper, we model the temporal relations between video moments by a two-dimensional map, where one dimension indicates the starting time of a moment and the other indicates the end time. This 2D temporal map can cover diverse video moments with different lengths, while representing their adjacent relations. Based on the 2D map, we propose a Temporal Adjacent Network (2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal relation, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed 2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our 2D-TAN outperforms the state-of-the-art.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑