arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.07516cs.CV
收藏

视频实例分割的离线到在线知识蒸馏

Offline-to-Online Knowledge Distillation for Video Instance Segmentation

DGIST(大邱庆北科学技术院)

Hojin Kim, Seunghun Lee, Sunghoon Im

摘要:

在本文中,我们提出了离线到在线的知识蒸馏(OOKD)的视频实例分割(VIS),它将丰富的视频知识从离线模型转移到在线模型进行一致性预测。与以往的方法,采用在线或离线模型,我们的单一在线模型通过提取离线知识,利用这两个模型。为了正确地传递知识,我们提出了查询过滤和关联(QFA),它将不相关的查询过滤到确切的实例。我们的KD与QFA增加了鲁棒性的特征匹配编码对象为中心的功能,从一个单一的帧补充的长期全球信息。我们还提出了一个简单的数据增强计划,知识蒸馏在VIS任务,公平地转移到在线模型的所有类的知识。大量的实验表明,我们的方法显着提高了视频实例分割的性能,特别是对于具有挑战性的数据集,包括长,动态序列。我们的方法在YTVIS-21、YTVIS-22和OVIS数据集上也达到了最先进的性能,mAP得分分别为46.1%、43.6%和31.1%。

英文摘要:

In this paper, we present offline-to-online knowledge distillation (OOKD) for video instance segmentation (VIS), which transfers a wealth of video knowledge from an offline model to an online model for consistent prediction. Unlike previous methods that having adopting either an online or offline model, our single online model takes advantage of both models by distilling offline knowledge. To transfer knowledge correctly, we propose query filtering and association (QFA), which filters irrelevant queries to exact instances. Our KD with QFA increases the robustness of feature matching by encoding object-centric features from a single frame supplemented by long-range global information. We also propose a simple data augmentation scheme for knowledge distillation in the VIS task that fairly transfers the knowledge of all classes into the online model. Extensive experiments show that our method significantly improves the performance in video instance segmentation, especially for challenging datasets including long, dynamic sequences. Our method also achieves state-of-the-art performance on YTVIS-21, YTVIS-22, and OVIS datasets, with mAP scores of 46.1%, 43.6%, and 31.1%, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑