Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Grounding
Hefei University of Technology (HFUT)(合肥工业大学);School of Computer Science and Information Engineering (School of Artificial Intelligence)(计算机与信息工程学院(人工智能学院));Intelligent Interconnected Systems Laboratory of Anhui Province(安徽省智能互联系统实验室)
Jingjing Hu, Dan Guo, Kun Li, Zhan Si, Xun Yang, Xiaojun Chang, Meng Wang
Inspired by the activity-silent and persistent activity mechanisms in human visual perception biology, we design a Unified Static and Dynamic Network (UniSDNet), to learn the semantic association between the video and text/audio queries in a cross-modal environment for efficient video grounding. For static modeling, we devise a novel residual structure (ResMLP) to boost the global comprehensive interaction between the video segments and queries, achieving more effective semantic enhancement/supplement. For dynamic modeling, we effectively exploit three characteristics of the persistent activity mechanism in our network design for a better video context comprehension. Specifically, we construct a diffusely connected video clip graph on the basis of 2D sparse temporal masking to reflect the "short-term effect" relationship. We innovatively consider the temporal distance and relevance as the joint "auxiliary evidence clues" and design a multi-kernel Temporal Gaussian Filter to expand the context clue into high-dimensional space, simulating the "complex visual perception", and then conduct element level filtering convolution operations on neighbour clip nodes in message passing stage for finally generating and ranking the candidate proposals. Our UniSDNet is applicable to both Natural Language Video Grounding (NLVG) and Spoken Language Video Grounding (SLVG) tasks. Our UniSDNet achieves SOTA performance on three widely used datasets for NLVG, as well as three datasets for SLVG, e.g., reporting new records at 38.88% R@1,IoU@0.7 on ActivityNet Captions and 40.26% R@1,IoU@0.5 on TACoS. To facilitate this field, we collect two new datasets (Charades-STA Speech and TACoS Speech) for SLVG task. Meanwhile, the inference speed of our UniSDNet is 1.56$\times$ faster than the strong multi-query benchmark. Code is available at: https://github.com/xian-sh/UniSDNet.
Static and Dynamic Graph Alignment Network for Temporal Video Grounding
Faculty of Electronic Engineering and Computer Science, Ningbo University(宁波大学电子工程与计算机科学学院);College of Computer Science, Inner Mongolia University(内蒙古大学计算机学院);College of Computing and Engineering, Hunan Normal University(湖南师范大学计算机与工程学院);Faculty of Computing, Georg-August-Universität Göttingen(哥廷根大学计算机学院);Center for Artificial Intelligence, Mathematical and Data Science, Nagoya University(名古屋大学人工智能、数学与数据科学中心)
HERO: Hierarchical Embedding-Refinement for Open-Vocabulary Temporal Sentence Grounding in Videos
Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University(电子科技大学复杂系统建模与仿真实验室,计算机科学与技术学院);Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University(浙江省空间信息感知与传输重点实验室,电子科技大学);Department of Psychological and Cognitive Sciences, Tsinghua University(清华大学心理与认知科学系)
Training-free Video Temporal Grounding using Large-scale Pre-trained Models
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);National Institute of Health Data Science, Peking University(北京大学健康数据科学国家研究院);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室)
Grounding is All You Need? Dual Temporal Grounding for Video Dialog
National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);Hefei University of Technology(合肥工业大学);Singapore Management University(新加坡管理大学)
Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);School of Software, Beihang University(北京航空航天大学软件学院);Meituan(美团);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程学院)
Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间安全学院)
EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer
Hefei University of Technology (HFUT)(合肥工业大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Anhui Zhonghuitong Technology Co., Ltd.(安徽中汇通科技有限公司)
ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding
Chongqing University(重庆大学);Tianjin University(天津大学);MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院MAIS);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore(新加坡科技研究局高性能计算研究所);Chongqing National Data AI Research Institute, AI Research Lab(重庆国家数据AI研究院,AI研究实验室)
ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models
Koç University(科奇大学);KUIS AI Center(KUIS人工智能中心);University of Pisa(比萨大学);Hacettepe University(哈塞特佩大学);Aselsan Research(阿塞尔桑研究);University of Malta(马耳他大学);Heidelberg University(海德堡大学);Amsterdam UMC, University of Amsterdam(阿姆斯特丹大学医学中心);Utrecht University(乌得勒支大学)
GMoT: Gated Motion-Aware Tokenization for Fine-Grained Micro-Gesture Video Reasoning with Multimodal LLMs
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Great Bay University(大湾区大学);Dongguan Key Laboratory for Intelligence and Information Technology(东莞市智能信息技术重点实验室)