arXiv 2403.114632024-05-14cs.CV

面向弱监督视频段落定位的联合对齐与回归孪生学习

Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding

School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（教育部机器智能与先进计算重点实验室）

Chaolei Tan, Jianhuang Lai, Wei-Shi Zheng, Jian-Fang Hu

备注：: Accepted to CVPR 2024. v2: fix a typo in figure 1

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有视频段落定位依赖大量时间标签的问题，提出带权重共享双分支的孪生学习框架SiamGTR，无需时间标注即可联合实现跨模态对齐与时间回归，在弱/半监督场景下性能超越现有最优方法。

摘要：

视频段落定位（Video Paragraph Grounding, VPG）是视频语言理解领域的新兴任务，旨在从未修剪的视频中定位出具有语义关联和时间顺序的多个句子。然而，现有VPG方法严重依赖大量时间标签，这类标签的获取耗时耗力。本文提出并探索弱监督视频段落定位（Weakly-Supervised Video Paragraph Grounding, WSVPG），以消除对时间标注的需求。与此前基于多示例学习或重建学习、采用两阶段候选排序的弱监督定位框架不同，我们提出一种新颖的孪生学习框架，在无时间戳标签的情况下联合学习跨模态特征对齐与时间坐标回归，实现WSVPG的简洁一阶段定位。具体而言，我们设计了孪生定位Transformer（Siamese Grounding TRansformer, SiamGTR），由两个权重共享的分支构成以学习互补监督信息：增强分支用于直接回归伪视频中完整段落的时间边界，推理分支则用于捕捉顺序引导的特征对应关系，以在普通视频中定位多个句子。我们通过大量实验证明，该范式在实现高效的弱监督或半监督学习方面具备出色的实用性与灵活性，性能优于采用同等或更强监督训练的现有最优方法。

英文摘要：

Video Paragraph Grounding (VPG) is an emerging task in video-language understanding, which aims at localizing multiple sentences with semantic relations and temporal order from an untrimmed video. However, existing VPG approaches are heavily reliant on a considerable number of temporal labels that are laborious and time-consuming to acquire. In this work, we introduce and explore Weakly-Supervised Video Paragraph Grounding (WSVPG) to eliminate the need of temporal annotations. Different from previous weakly-supervised grounding frameworks based on multiple instance learning or reconstruction learning for two-stage candidate ranking, we propose a novel siamese learning framework that jointly learns the cross-modal feature alignment and temporal coordinate regression without timestamp labels to achieve concise one-stage localization for WSVPG. Specifically, we devise a Siamese Grounding TRansformer (SiamGTR) consisting of two weight-sharing branches for learning complementary supervision. An Augmentation Branch is utilized for directly regressing the temporal boundaries of a complete paragraph within a pseudo video, and an Inference Branch is designed to capture the order-guided feature correspondence for localizing multiple sentences in a normal video. We demonstrate by extensive experiments that our paradigm has superior practicability and flexibility to achieve efficient weakly-supervised or semi-supervised learning, outperforming state-of-the-art methods trained with the same or stronger supervision.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。