利用时间关系的自然语言视频时刻定位
Exploiting Temporal Relationships in Video Moment Localization with Natural Language
摘要:
我们解决的问题,视频时刻定位与自然语言,即本地化的视频片段描述的自然语言的句子。虽然大多数先前的工作集中在接地查询作为一个整体,时间依赖性和推理的事件之间的文本没有得到充分考虑。在本文中,我们提出了一种新的时间组合模块化网络(TCMN)的树注意力网络首先自动分解成三个描述的主要事件,上下文事件和时间信号的句子。然后利用两个模块来测量每个片段与分解的描述之间的视觉相似性和位置相似性。此外,由于主事件和上下文事件可能依赖于不同的模态(RGB或光流),因此我们使用后期融合来形成四个模型的集合,其中每个模型都由视觉输入的一个组合独立训练。实验表明,我们的模型在TEMPO数据集上的性能优于最先进的方法。
英文摘要:
We address the problem of video moment localization with natural language, i.e. localizing a video segment described by a natural language sentence. While most prior work focuses on grounding the query as a whole, temporal dependencies and reasoning between events within the text are not fully considered. In this paper, we propose a novel Temporal Compositional Modular Network (TCMN) where a tree attention network first automatically decomposes a sentence into three descriptions with respect to the main event, context event and temporal signal. Two modules are then utilized to measure the visual similarity and location similarity between each segment and the decomposed descriptions. Moreover, since the main event and context event may rely on different modalities (RGB or optical flow), we use late fusion to form an ensemble of four models, where each model is independently trained by one combination of the visual input. Experiments show that our model outperforms the state-of-the-art methods on the TEMPO dataset.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。