We address the problem of retrieving a specific moment from an untrimmed video by natural language. It is a challenging problem because a target moment may take place in the context of other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they do not fully consider the temporal contexts between temporal moments. In this paper, we model the temporal context between video moments by a set of predefined two-dimensional maps under different temporal scales. For each map, one dimension indicates the starting time of a moment and the other indicates the duration. These 2D temporal maps can cover diverse video moments with different lengths, while representing their adjacent contexts at different temporal scales. Based on the 2D temporal maps, we propose a Multi-Scale Temporal Adjacent Network (MS-2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal contexts at each scale, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed MS-2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our MS-2D-TAN outperforms the state of the art.
Shandong Jianzhu University(山东建筑大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Shandong University(山东大学);Hefei University of Technology(合肥工业大学);Lenovo Company Ltd.(联想集团有限公司)
MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction
Nanyang Technological University(南洋理工大学);Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);Centre for Frontier AI Research, A*STAR(新加坡科技研究局前沿人工智能研究中心)
DORi: Discovering Object Relationship for Moment Localization of a Natural-Language Query in Video
Australian National University(澳大利亚国立大学);Australian Centre for Robotic Vision (ACRV)(澳大利亚机器人视觉中心(ACRV));Graduate School of Engineering, The University of Tokyo(东京大学工学研究科);A*AI, A*STAR Singapore(A*AI,新加坡A*STAR)
Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Shenzhen H&T Intelligent Control Co., Ltd.(深圳H&T智能控制有限公司);Southern University of Science and Technology(南方科技大学);Southeast University(东南大学)
Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention
Australian National University(澳大利亚国立大学);Australian Centre for Robotic Vision (ACRV)(澳大利亚机器人视觉中心(ACRV));Graduate School of Engineering, The University of Tokyo(东京大学工学研究生院);The University of Tokyo(东京大学);Graduate School of Engineering(东京大学工学研究生院)
FPV-CRASH: A Dataset for Offline Crash Moment Localization in Drone First-Person-View Videos
School of Communication Engineering, Hangzhou Dianzi University(杭州电子科技大学通信工程学院);School of Computer Science, Faculty of Science, University of Auckland(奥克兰大学理学院计算机科学系);Department of Computer and Information Science, Faculty of Science and Technology, University of Macau(澳门科技大学理学院计算机与信息科学系)
A Multimodal Deviation Perceiving Framework for Weakly-Supervised Temporal Forgery Localization
School of Computer Science and Engineering, Sun Yat-sen University(计算机科学与工程学院,中山大学);State Key Laboratory of Mathematical Engineering and Advanced Computing(数学工程与先进计算国家重点实验室);School of Cyber Science and Engineering, Wuhan University(网络科学与工程学院,武汉大学)
Hypotheses Tree Building for One-Shot Temporal Sentence Localization
Huazhong University of Science and Technology(华中科技大学);Peking University(北京大学);Nanyang Technological University(南洋理工大学);ProtagoLabs Inc(ProtagoLabs公司);Tsinghua University(清华大学);Microsoft Research(微软研究院)
Multi-Scale Context-Guided Lumbar Spine Disease Identification with Coarse-to-fine Localization and Classification
Center for Data Science, Peking University(北京大学数据科学中心);National Engineering Laboratory for Big Data Analysis and Applications, Peking University(北京大学大数据分析与应用国家工程实验室);National Biomedical Imaging Center, Peking University(北京大学生物医学成像国家中心)
Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment Localization
Huazhong University of Science and Technology(华中科技大学);Huawei Cloud(华为云);Columbia University(哥伦比亚大学);Zhejiang Gongshang University(浙江工商大学);Dalian University of Technology(大连理工大学)