arXiv 2404.135052024-04-21cs.CV

Dynamic in Static：用于自监督视频目标分割的混合视觉对应

Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentation

Nanjing University of Science and Technology（南京理工大学）；Zhejiang University（浙江大学）；University of Birmingham（伯明翰大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））

Gensheng Pei, Yazhou Yao, Jianbo Jiao, Wenguan Wang, Liqiang Nie, Jinhui Tang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出HVC框架，通过从静态图像提取伪动态信号并利用全卷积架构学习静态-动态视觉对应，仅需约2小时训练即在多个自监督视频目标分割基准上达到SOTA性能。

摘要：

传统的视频目标分割（VOS）方法通常需要大量像素级标注的视频数据进行全监督学习。本文提出了HVC，一个用于自监督VOS的混合静态-动态视觉对应框架。HVC从静态图像中提取伪动态信号，实现了高效且可扩展的VOS模型。该方法采用极简的全卷积架构，在图像裁剪视图中捕捉静态-动态视觉对应。为此，我们提出了一种统一的自监督方法来学习静态-动态特征相似性的视觉表示。首先，利用裁剪视图间的先验坐标信息建立静态对应，引导形成一致的静态特征表示。随后，设计简洁的卷积层捕捉两个视图间的前向/后向伪动态信号，作为动态表示的线索。最后，提出混合视觉对应损失来学习静态与动态联合一致性表示。该方法无需复杂技巧，仅需使用静态图像数据进行一次训练，显著降低了内存消耗（约16GB）和训练时间（约2小时）。此外，HVC在多个自监督VOS基准和额外的视频标签传播任务中均取得了最先进的性能。

英文摘要：

Conventional video object segmentation (VOS) methods usually necessitate a substantial volume of pixel-level annotated video data for fully supervised learning. In this paper, we present HVC, a \textbf{h}ybrid static-dynamic \textbf{v}isual \textbf{c}orrespondence framework for self-supervised VOS. HVC extracts pseudo-dynamic signals from static images, enabling an efficient and scalable VOS model. Our approach utilizes a minimalist fully-convolutional architecture to capture static-dynamic visual correspondence in image-cropped views. To achieve this objective, we present a unified self-supervised approach to learn visual representations of static-dynamic feature similarity. Firstly, we establish static correspondence by utilizing a priori coordinate information between cropped views to guide the formation of consistent static feature representations. Subsequently, we devise a concise convolutional layer to capture the forward / backward pseudo-dynamic signals between two views, serving as cues for dynamic representations. Finally, we propose a hybrid visual correspondence loss to learn joint static and dynamic consistency representations. Our approach, without bells and whistles, necessitates only one training session using static image data, significantly reducing memory consumption ($\sim$16GB) and training time ($\sim$\textbf{2h}). Moreover, HVC achieves state-of-the-art performance in several self-supervised VOS benchmarks and additional video label propagation tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。