arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.00684cs.CV
收藏

静态与动态图对齐网络用于时序视频定位

Static and Dynamic Graph Alignment Network for Temporal Video Grounding

Faculty of Electronic Engineering and Computer Science, Ningbo University(宁波大学电子工程与计算机科学学院);College of Computer Science, Inner Mongolia University(内蒙古大学计算机学院);College of Computing and Engineering, Hunan Normal University(湖南师范大学计算机与工程学院);Faculty of Computing, Georg-August-Universität Göttingen(哥廷根大学计算机学院);Center for Artificial Intelligence, Mathematical and Data Science, Nagoya University(名古屋大学人工智能、数学与数据科学中心)

Zhanjie Hu, Bolin Zhang, Jianhua Wang, Jianbo Zheng, Chenchen Yan, Takahiro Komamizu, Ichiro Ide, Jiangbo Qian

AI总结:

本文提出SDGAN,通过联合静态和动态视觉特征构建互补的时序图,引入查询-片段对比学习和自适应图建模,结合多粒度时间提案和渐进易难训练策略,提升时序视频定位的性能。

摘要:

时序视频定位(TVG)旨在定位未剪辑视频中与给定自然语言查询语义对应的时间点。最近,图卷积网络(GCN)被广泛应用于TVG中,以建模视频片段间的时序关系并通过构建片段级图来增强上下文推理。尽管其有效性,现有基于GCN的TVG方法面临三个关键瓶颈:1)大多数方法仅使用静态或动态特征构建图节点,导致视觉表示不完整且忽略互补语义;2)大多数方法以查询无关的方式构建时序图,导致时序图表示内的特征交互低效;3)大多数方法常面临单一粒度的语义匹配,直接在复杂的时序定位任务上训练可能导致收敛缓慢和精度不佳。为解决这些挑战,我们提出静态与动态图对齐网络(SDGAN)。首先,SDGAN联合利用静态和动态视觉特征构建两个互补的时序图,并执行位置级节点对齐,从而获得更具表达力和鲁棒性的视觉表示。其次,SDGAN引入查询-片段对比学习和自适应图建模,以显式对齐视觉片段与其对应的文本查询,获得查询感知的视觉表示。第三,SDGAN结合多粒度时间提案在渐进易难训练策略中,有效连接粗粒度语义定位和细粒度时间边界细化。在三个基准数据集上的广泛实验表明,SDGAN在复杂的TVG场景中实现了优越的性能。代码和数据集可在https://github.com/ZhanJieHu/SDGAN获取。

英文摘要:

Temporal Video Grounding (TVG) aims to localize temporal moments in an untrimmed video that semantically correspond to given natural language queries. Recently, Graph Convolutional Networks (GCN) have been widely adopted in TVG to model temporal relations among video clips and enhance contextual reasoning by constructing clip-level graphs. Despite their effectiveness, existing GCN-based TVG methods encounter three critical bottlenecks: 1) Most methods construct graph nodes using either static or dynamic features alone, resulting in incomplete visual representation and overlooking complementary semantics, 2) Most methods construct temporal graphs in a query-agnostic manner, leading to inefficient feature interaction within the temporal graph representation, and 3) Most methods often suffer from a single-granularity semantic matching, while direct training on complex temporal localization task may lead to slow convergence and suboptimal precision. To address these challenges, we propose Static and Dynamic Graph Alignment Network (SDGAN). First, SDGAN jointly exploits static and dynamic visual features to construct two complementary temporal graphs and performs Position-wise Nodes Alignment, enabling more expressive and robust visual representation. Second, SDGAN introduces Query-Clip Contrastive Learning and Adaptive Graph Modeling to explicitly align visual clips with their corresponding textual queries, yielding query-aware visual representations. Third, SDGAN incorporates multi-granularity temporal proposals within Progressive Easy-to-Hard Training Strategy, effectively bridging coarse-grained semantic localization and fine-grained temporal boundary refinement. Extensive experiments on three benchmark datasets demonstrate that SDGAN achieves superior performance across complex TVG scenarios. Codes and datasets are available at https://github.com/ZhanJieHu/SDGAN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑