arXiv 2410.013412024-12-23cs.CV

面向文本监督的第一人称语义分割的认知迁移与解耦

Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation

School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）

Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Fanman Meng, Qingbo Wu, Hongliang Li

备注：: Accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)
DOI：: 10.1109/TCSVT.2024.3521955

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CTDN网络，通过认知迁移与前景-背景解耦解决文本监督第一人称语义分割中的关系不敏感和物体干扰问题，在多个基准上显著超越现有方法。

摘要：

本文探索了一种新颖的文本监督第一人称语义分割（TESS）任务，该任务旨在利用图像级标签中的文本对第一人称图像进行弱监督，从而为像素级类别分配标签。在这一具有广阔前景的任务中，第一人称场景包含密集的穿戴者-物体关系以及物体间干扰。然而，大多数近期的第三人称视角方法利用冻结的对比语言-图像预训练（CLIP）模型，该模型在面向语义的第三人称数据上预训练，由于“关系不敏感”问题在第一人称视角下表现不佳。因此，我们提出了一种认知迁移与解耦网络（CTDN），该网络首先通过关联图像和文本来学习第一人称穿戴者-物体关系。此外，开发了一个认知迁移模块（CTM），从大规模预训练模型中蒸馏认知知识到我们的模型，以识别具有各种语义的第一人称物体。基于迁移的认知，前景-背景解耦模块（FDM）解耦视觉表示，以显式区分前景和背景区域，从而减轻在第一人称关系学习过程中由前景-背景干扰物体引起的错误激活区域。在四个TESS基准上的大量实验证明了我们方法的有效性，其性能大幅优于许多近期的相关方法。代码将发布于 https://github.com/ZhaofengSHI/CTDN。

英文摘要：

In this paper, we explore a novel Text-supervised Egocentic Semantic Segmentation (TESS) task that aims to assign pixel-level categories to egocentric images weakly supervised by texts from image-level labels. In this task with prospective potential, the egocentric scenes contain dense wearer-object relations and inter-object interference. However, most recent third-view methods leverage the frozen Contrastive Language-Image Pre-training (CLIP) model, which is pre-trained on the semantic-oriented third-view data and lapses in the egocentric view due to the ``relation insensitive" problem. Hence, we propose a Cognition Transferring and Decoupling Network (CTDN) that first learns the egocentric wearer-object relations via correlating the image and text. Besides, a Cognition Transferring Module (CTM) is developed to distill the cognitive knowledge from the large-scale pre-trained model to our model for recognizing egocentric objects with various semantics. Based on the transferred cognition, the Foreground-background Decoupling Module (FDM) disentangles the visual representations to explicitly discriminate the foreground and background regions to mitigate false activation areas caused by foreground-background interferential objects during egocentric relation learning. Extensive experiments on four TESS benchmarks demonstrate the effectiveness of our approach, which outperforms many recent related methods by a large margin. Code will be available at https://github.com/ZhaofengSHI/CTDN.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。