解耦静态与分层运动感知用于指代表达视频分割
Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation
AI总结：
针对指代表达视频分割中静态与运动线索混淆的问题，本文提出表达解耦、分层运动感知和对比学习方法，在五个数据集上达到最先进性能。
摘要：
指代表达视频分割依赖自然语言表达来识别并分割物体，通常强调运动线索。以往工作将句子视为整体，并直接在视频级执行识别，从而将静态图像级线索与时间运动线索混在一起。然而，图像级特征无法很好地理解句子中的运动线索，而且静态线索对时间感知并不关键。事实上，静态线索有时会因掩盖运动线索而干扰时间感知。本文提出将视频级指代表达理解解耦为静态感知和运动感知，并特别强调增强时间理解。首先，我们引入表达解耦模块，使静态线索和运动线索发挥各自不同的作用，缓解句子嵌入忽视运动线索的问题。其次，我们提出分层运动感知模块，以在不同时间尺度上有效捕获时间信息。此外，我们采用对比学习来区分视觉相似物体的运动。这些贡献在五个数据集上取得了最先进性能，包括在具有挑战性的 MeViS 数据集上实现显著的 9.2% J&F 提升。代码可在 https://github.com/heshuting555/DsHmp 获取。
英文摘要：
Referring video segmentation relies on natural language expressions to identify and segment objects, often emphasizing motion clues. Previous works treat a sentence as a whole and directly perform identification at the video-level, mixing up static image-level cues with temporal motion cues. However, image-level features cannot well comprehend motion cues in sentences, and static cues are not crucial for temporal perception. In fact, static cues can sometimes interfere with temporal perception by overshadowing motion cues. In this work, we propose to decouple video-level referring expression understanding into static and motion perception, with a specific emphasis on enhancing temporal comprehension. Firstly, we introduce an expression-decoupling module to make static cues and motion cues perform their distinct role, alleviating the issue of sentence embeddings overlooking motion cues. Secondly, we propose a hierarchical motion perception module to capture temporal information effectively across varying timescales. Furthermore, we employ contrastive learning to distinguish the motions of visually similar objects. These contributions yield state-of-the-art performance across five datasets, including a remarkable $\textbf{9.2%}$ $\mathcal{J\&F}$ improvement on the challenging $\textbf{MeViS}$ dataset. Code is available at https://github.com/heshuting555/DsHmp.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
