arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2201.08071cs.CVcs.AIcs.CLcs.MM

视频中的时序句子定位:综述与未来方向

Temporal Sentence Grounding in Videos: A Survey and Future Directions

Nanyang Technological University(南洋理工大学);Alibaba Group(阿里巴巴集团);Centre for Frontier AI Research, A*STAR(新加坡科技研究局前沿人工智能研究中心)

Hao Zhang, Aixin Sun, Wei Jing, Joey Tianyi Zhou

AI总结:

本文综述视频时序句子定位(TSGV)的通用流程、多模态理解与交互方法及技术分类,分析现有问题并指出未来研究方向。

摘要:

视频中的时序句子定位(TSGV),又称自然语言视频定位(NLVL)或视频瞬间检索(VMR),旨在从未经裁剪的视频中检索出与语言查询在语义上相对应的时序片段。TSGV连接计算机视觉与自然语言,已引起两个领域研究者的广泛关注。本综述试图总结TSGV的基础概念、当前研究现状以及未来研究方向。作为背景,我们以教程式风格介绍TSGV功能组件的通用结构:从原始视频和语言查询的特征提取,到目标片段的答案预测。随后,我们综述用于多模态理解与交互的技术,这是TSGV的核心重点,用于实现两种模态之间的有效对齐。我们构建了TSGV技术的分类体系,并详细阐述不同类别中的方法及其优势与不足。最后,我们讨论当前TSGV研究存在的问题,并分享关于有前景研究方向的见解。

英文摘要:

Temporal sentence grounding in videos (TSGV), \aka natural language video localization (NLVL) or video moment retrieval (VMR), aims to retrieve a temporal moment that semantically corresponds to a language query from an untrimmed video. Connecting computer vision and natural language, TSGV has drawn significant attention from researchers in both communities. This survey attempts to provide a summary of fundamental concepts in TSGV and current research status, as well as future research directions. As the background, we present a common structure of functional components in TSGV, in a tutorial style: from feature extraction from raw video and language query, to answer prediction of the target moment. Then we review the techniques for multimodal understanding and interaction, which is the key focus of TSGV for effective alignment between the two modalities. We construct a taxonomy of TSGV techniques and elaborate the methods in different categories with their strengths and weaknesses. Lastly, we discuss issues with the current TSGV research and share our insights about promising research directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑