arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.12480cs.CV
收藏

IDPro:通过ID查询并发传播实现灵活的交互式视频目标分割

IDPro: Flexible Interactive Video Object Segmentation by ID-queried Concurrent Propagation

College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Harvard University(哈佛大学)

Kexin Li, Tao Jiang, Zongxin Yang, Yi Yang, Yueting Zhuang, Jun Xiao

AI总结:

本文提出IDPro框架,通过多帧同时交互、跨帧协同交互模块和ID查询机制,实现灵活高效的交互式视频目标分割,在DAVIS 2017上取得最优性能且速度显著提升。

摘要:

交互式视频目标分割(iVOS)是一项需要实时人机交互的挑战性任务。为了改善用户体验,必须考虑用户的输入习惯、分割质量、运行时间和内存消耗。然而,现有方法因单一输入模式和较慢的运行速度而损害了用户体验。具体而言,这些方法仅允许用户与单个帧进行交互,这限制了用户意图的表达。为了克服这些局限性并更好地契合人们的使用习惯,我们提出了一种能够同时接受多个帧并探索跨帧协同交互(SIAF)的框架。具体地,我们设计了跨帧交互模块(Across-Frame Interaction Module),使用户能够在多个帧上自由标注不同目标。AFI模块将在多个交互帧之间迁移涂鸦信息,并生成多帧掩码。此外,我们采用ID查询机制来批量处理多个目标。进一步地,为了实现更高效的传播和轻量化模型,我们设计了一种截断式重传播策略来替代以往的多轮融合模块,该策略采用跨轮记忆来存储重要的交互信息。我们的SwinB-SIAF在DAVIS 2017上取得了新的最优性能(89.6%,J&F@60)。此外,在具有挑战性的多目标场景下,我们的R50-SIAF比最先进的竞争对手快3倍以上。

英文摘要:

Interactive Video Object Segmentation (iVOS) is a challenging task that requires real-time human-computer interaction. To improve the user experience, it is important to consider the user's input habits, segmentation quality, running time and memory consumption.However, existing methods compromise user experience with single input mode and slow running speed. Specifically, these methods only allow the user to interact with one single frame, which limits the expression of the user's intent.To overcome these limitations and better align with people's usage habits, we propose a framework that can accept multiple frames simultaneously and explore synergistic interaction across frames (SIAF). Concretely, we designed the Across-Frame Interaction Module that enables users to annotate different objects freely on multiple frames. The AFI module will migrate scribble information among multiple interactive frames and generate multi-frame masks. Additionally, we employ the id-queried mechanism to process multiple objects in batches. Furthermore, for a more efficient propagation and lightweight model, we design a truncated re-propagation strategy to replace the previous multi-round fusion module, which employs an across-round memory that stores important interaction information. Our SwinB-SIAF achieves new state-of-the-art performance on DAVIS 2017 (89.6%, J&F@60). Moreover, our R50-SIAF is more than 3 faster than the state-of-the-art competitor under challenging multi-object scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑