arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1904.02755cs.CL
收藏

ExCL:使用自然语言描述的提取剪辑定位

ExCL: Extractive Clip Localization Using Natural Language Descriptions

Carnegie Mellon University(卡内基梅隆大学)

Soham Ghosh, Anuva Agarwal, Zarana Parekh, Alexander Hauptmann

摘要:

基于给定的自然语言查询检索视频内的剪辑的任务需要在多个帧上进行跨模态推理。以前的方法,如滑动窗口分类器是低效的,而文本剪辑相似性驱动的基于排名的方法,如段建议网络要复杂得多。为了选择与给定文本描述对应的最相关的视频片段,我们提出了一种新的提取方法,该方法通过利用文本和视频之间的跨模态交互来预测开始和结束帧-这消除了检索和重新排名多个提案片段的需要。使用递归网络,我们将这两种模态编码为联合表示,然后将其用于开始-结束帧预测器网络的不同变体中。通过广泛的实验和消融分析,我们证明了我们简单而优雅的方法在两个数据集上的性能显着优于最先进的技术,并且在第三个数据集上具有相当的性能。

英文摘要:

The task of retrieving clips within videos based on a given natural language query requires cross-modal reasoning over multiple frames. Prior approaches such as sliding window classifiers are inefficient, while text-clip similarity driven ranking-based approaches such as segment proposal networks are far more complicated. In order to select the most relevant video clip corresponding to the given text description, we propose a novel extractive approach that predicts the start and end frames by leveraging cross-modal interactions between the text and video - this removes the need to retrieve and re-rank multiple proposal segments. Using recurrent networks we encode the two modalities into a joint representation which is then used in different variants of start-end frame predictor networks. Through extensive experimentation and ablative analysis, we demonstrate that our simple and elegant approach significantly outperforms state of the art on two datasets and has comparable performance on a third.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑