arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.14174cs.CV
收藏

统一静态与动态网络:面向视频定位的高效时序滤波

Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Grounding

Hefei University of Technology (HFUT)(合肥工业大学);School of Computer Science and Information Engineering (School of Artificial Intelligence)(计算机与信息工程学院(人工智能学院));Intelligent Interconnected Systems Laboratory of Anhui Province(安徽省智能互联系统实验室)

Jingjing Hu, Dan Guo, Kun Li, Zhan Si, Xun Yang, Xiaojun Chang, Meng Wang

AI总结:

本文提出统一静态与动态网络UniSDNet,通过残差MLP和多核时序高斯滤波实现视频与文本/音频查询的高效跨模态语义关联,在NLVG和SLVG任务上取得SOTA性能并提升推理速度。

摘要:

受人类视觉感知生物学中活动静默与持续活动机制的启发,我们设计了一个统一静态与动态网络(UniSDNet),用于在跨模态环境中学习视频与文本/音频查询之间的语义关联,以实现高效的视频定位。在静态建模方面,我们设计了一种新颖的残差结构(ResMLP),以增强视频片段与查询之间的全局综合交互,从而实现更有效的语义增强/补充。在动态建模方面,我们在网络设计中有效利用了持续活动机制的三个特性,以更好地理解视频上下文。具体来说,我们基于2D稀疏时序掩码构建了一个扩散连接的视频片段图,以反映“短期效应”关系。我们创新性地将时序距离和相关性视为联合“辅助证据线索”,并设计了一个多核时序高斯滤波器,将上下文线索扩展到高维空间,模拟“复杂视觉感知”,然后在消息传递阶段对邻居片段节点进行元素级滤波卷积操作,最终生成并对候选提议进行排序。我们的UniSDNet同时适用于自然语言视频定位(NLVG)和口语语言视频定位(SLVG)任务。我们的UniSDNet在三个广泛使用的NLVG数据集以及三个SLVG数据集上均取得了SOTA性能,例如,在ActivityNet Captions上报告了38.88% R@1,IoU@0.7的新纪录,在TACoS上报告了40.26% R@1,IoU@0.5的新纪录。为了促进该领域的发展,我们为SLVG任务收集了两个新数据集(Charades-STA Speech和TACoS Speech)。同时,我们的UniSDNet的推理速度比强多查询基准快1.56倍。代码可在以下地址获取:https://github.com/xian-sh/UniSDNet。

英文摘要:

Inspired by the activity-silent and persistent activity mechanisms in human visual perception biology, we design a Unified Static and Dynamic Network (UniSDNet), to learn the semantic association between the video and text/audio queries in a cross-modal environment for efficient video grounding. For static modeling, we devise a novel residual structure (ResMLP) to boost the global comprehensive interaction between the video segments and queries, achieving more effective semantic enhancement/supplement. For dynamic modeling, we effectively exploit three characteristics of the persistent activity mechanism in our network design for a better video context comprehension. Specifically, we construct a diffusely connected video clip graph on the basis of 2D sparse temporal masking to reflect the "short-term effect" relationship. We innovatively consider the temporal distance and relevance as the joint "auxiliary evidence clues" and design a multi-kernel Temporal Gaussian Filter to expand the context clue into high-dimensional space, simulating the "complex visual perception", and then conduct element level filtering convolution operations on neighbour clip nodes in message passing stage for finally generating and ranking the candidate proposals. Our UniSDNet is applicable to both Natural Language Video Grounding (NLVG) and Spoken Language Video Grounding (SLVG) tasks. Our UniSDNet achieves SOTA performance on three widely used datasets for NLVG, as well as three datasets for SLVG, e.g., reporting new records at 38.88% R@1,IoU@0.7 on ActivityNet Captions and 40.26% R@1,IoU@0.5 on TACoS. To facilitate this field, we collect two new datasets (Charades-STA Speech and TACoS Speech) for SLVG task. Meanwhile, the inference speed of our UniSDNet is 1.56$\times$ faster than the strong multi-query benchmark. Code is available at: https://github.com/xian-sh/UniSDNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑