arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.02646cs.CV
收藏

基于多尺度二维时间邻接网络的自然语言矩定位

Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural Language

University of Rochester(罗切斯特大学);Microsoft(微软)

Songyang Zhang, Houwen Peng, Jianlong Fu, Yijuan Lu, Jiebo Luo

摘要:

我们解决的问题,检索一个特定的时刻,从一个未经修剪的视频由自然语言。这是一个具有挑战性的问题,因为目标时刻可能发生在未修剪视频中的其他时间时刻的上下文中。现有的方法不能很好地解决这一挑战,因为它们没有充分考虑时间时刻之间的时间上下文。在本文中,我们通过一组预定义的二维地图在不同的时间尺度下的视频时刻之间的时间上下文建模。对于每个地图,一个维度表示时刻的开始时间,另一个维度表示持续时间。这些2D时间图可以覆盖具有不同长度的不同视频时刻,同时以不同的时间尺度表示它们的相邻上下文。基于二维时间地图,我们提出了一个多尺度时间相邻网络(MS-2D-TAN),一个单镜头框架的时刻定位。它能够在每个尺度上对相邻的时间上下文进行编码,同时学习用于将视频时刻与参考表达式相匹配的判别特征。我们在三个具有挑战性的基准上评估了拟议的MS-2D-TAN,即,Charades-STA、ActivityNet Captions和TACoS,其中我们的MS-2D-TAN优于最先进的技术。

英文摘要:

We address the problem of retrieving a specific moment from an untrimmed video by natural language. It is a challenging problem because a target moment may take place in the context of other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they do not fully consider the temporal contexts between temporal moments. In this paper, we model the temporal context between video moments by a set of predefined two-dimensional maps under different temporal scales. For each map, one dimension indicates the starting time of a moment and the other indicates the duration. These 2D temporal maps can cover diverse video moments with different lengths, while representing their adjacent contexts at different temporal scales. Based on the 2D temporal maps, we propose a Multi-Scale Temporal Adjacent Network (MS-2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal contexts at each scale, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed MS-2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our MS-2D-TAN outperforms the state of the art.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑