In this paper, we address the unexplored question of temporal sentence localization in human motions (TSLM), aiming to locate a target moment from a 3D human motion that semantically corresponds to a text query. Considering that 3D human motions are captured using specialized motion capture devices, motions with only a few joints lack complex scene information like objects and lighting. Due to this character, motion data has low contextual richness and semantic ambiguity between frames, which limits the accuracy of predictions made by current video localization frameworks extended to TSLM to only a rough level. To refine this, we devise two novel label-prior-assisted training schemes: one embed prior knowledge of foreground and background to highlight the localization chances of target moments, and the other forces the originally rough predictions to overlap with the more accurate predictions obtained from the flipped start/end prior label sequences during recovery training. We show that injecting label-prior knowledge into the model is crucial for improving performance at high IoU. In our constructed TSLM benchmark, our model termed MLP achieves a recall of 44.13 at IoU@0.7 on the BABEL dataset and 71.17 on HumanML3D (Restore), outperforming prior works. Finally, we showcase the potential of our approach in corpus-level moment retrieval. Our source code is openly accessible at https://github.com/eanson023/mlp.
To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression
Tsinghua-Berkeley Shenzhen Institute, Tsinghua University(清华大学清华-伯克利深圳学院);JD AI Research(京东人工智能研究院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization
School of Electrical and Information Engineering, University of Sydney(悉尼大学电气与信息工程学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment Localization
Huazhong University of Science and Technology(华中科技大学);Huawei Cloud(华为云);Columbia University(哥伦比亚大学);Zhejiang Gongshang University(浙江工商大学);Dalian University of Technology(大连理工大学)
Boosting Temporal Sentence Grounding via Causal Inference
School of Electronic Engineering, Xidian University Xi'an China;School of Information Science \& Engineering, Lanzhou University Lanzhou China;Xidian University;Lanzhou University
Hierarchical Local-Global Transformer for Temporal Sentence Grounding
Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology(大数据安全湖北工程研究中心,华中科技大学网络安全科学与工程学院);Wangxuan Institute of Computer fTechnology, Peking University(王宣计算机技术研究院,北京大学);School of software, Dalian University of Technology(软件学院,大连理工大学);School of Computer Science, and Technology, Huazhong University of Science, and Technology(计算机科学与技术学院,华中科技大学)
DORi: Discovering Object Relationship for Moment Localization of a Natural-Language Query in Video
Australian National University(澳大利亚国立大学);Australian Centre for Robotic Vision (ACRV)(澳大利亚机器人视觉中心(ACRV));Graduate School of Engineering, The University of Tokyo(东京大学工学研究科);A*AI, A*STAR Singapore(A*AI,新加坡A*STAR)
What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions
Columbia University(哥伦比亚大学);Goethe University Frankfurt(法兰克福大学);University of Bonn(波恩大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Quality Match GmbH(Quality Match有限公司);IBM Research AI(IBM研究院人工智能部门);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention
Australian National University(澳大利亚国立大学);Australian Centre for Robotic Vision (ACRV)(澳大利亚机器人视觉中心(ACRV));Graduate School of Engineering, The University of Tokyo(东京大学工学研究生院);The University of Tokyo(东京大学);Graduate School of Engineering(东京大学工学研究生院)
Context-aware Biaffine Localizing Network for Temporal Sentence Grounding
Huazhong University of Science and Technology(华中科技大学);Zhejiang Gongshang University(浙江工商大学);Microsoft AI(微软人工智能);Dalian University of Technology(大连理工大学);Huawei Cloud(华为云)
Hypotheses Tree Building for One-Shot Temporal Sentence Localization
Huazhong University of Science and Technology(华中科技大学);Peking University(北京大学);Nanyang Technological University(南洋理工大学);ProtagoLabs Inc(ProtagoLabs公司);Tsinghua University(清华大学);Microsoft Research(微软研究院)
Rethinking the Video Sampling and Reasoning Strategies for Temporal Sentence Grounding
ProtagoLabs Inc(ProtagoLabs公司);Microsoft Research(微软研究院);Beijing Institute of Technology(北京理工大学);Huazhong University of Science and Technology(华中科技大学);Sichuan University(四川大学);Dalian University of Technology(大连理工大学);Lehigh University(利哈伊大学);Peking University(北京大学)
You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos
The Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology(大数据安全湖北工程研究中心,网络安全科学与工程学院,华中科技大学);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学)
Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间安全学院)
Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language
Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science of Technology(湖北大数据安全工程研究中心,网络安全学院,华中科技大学);Peking University(北京大学);Henan University(河南大学);Dalian University of Technology(大连理工大学);Sichuan University(四川大学);Shenzhen University(深圳大学);Huazhong University of Science and Technology(华中科技大学)
Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining
University of Science and Technology of China(中国科学技术大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Beihang University(北京航空航天大学);State Key Laboratory for Novel Software Technology(国家软件创新科技重点实验室);Nanjing University(南京大学);Shenzhen Key Laboratory of Computer Vision and Pattern Recognition(深圳计算机视觉与模式识别重点实验室);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Chinese Academy of Sciences(中国科学院)
Learning Sample Importance for Cross-Scenario Video Temporal Grounding
Peking University(北京大学)
时间接地的任务旨在定位视频时刻在未经修剪的视频,与给定的句子查询。本文首次研究了时间背景任务中的一些表面偏差,并提出了一种新的有针对性的解决方案。最令人震惊的是,我们观察到现有的时间地面模型严重依赖于一些偏见(例如,对频繁概念或某些时间间隔的高偏好)。这导致在跨场景测试设置中推广模型时性能较差。为此,我们提出了一种新的方法称为DebiasTLL(DebiasTemporal Language Localizer),以防止模型天真地记住偏见,并强制它基于真正的模态间关系来建立查询语句。Debias-TLL同时训练两个模型。通过我们的设计,当判断样本时,这两个模型的预测值存在较大差异时,显示出更高的偏倚样本概率。利用信息的差异,我们设计了一个数据重新加权计划,以减轻数据偏差。我们评估所提出的模型在跨场景的时间接地,其中的训练/测试数据是异质源。实验结果表明,与现有的竞争对手相比,该方法具有较大的优势。