Applying the knowledge of an object detector trained on a specific domain directly onto a new domain is risky, as the gap between two domains can severely degrade model's performance. Furthermore, since different instances commonly embody distinct modal information in object detection scenario, the feature alignment of source and target domain is hard to be realized. To mitigate these problems, we propose a Graph-induced Prototype Alignment (GPA) framework to seek for category-level domain alignment via elaborate prototype representations. In the nutshell, more precise instance-level features are obtained through graph-based information propagation among region proposals, and, on such basis, the prototype representation of each class is derived for category-level domain alignment. In addition, in order to alleviate the negative effect of class-imbalance on domain adaptation, we design a Class-reweighted Contrastive Loss to harmonize the adaptation training process. Combining with Faster R-CNN, the proposed framework conducts feature alignment in a two-stage manner. Comprehensive results on various cross-domain detection tasks demonstrate that our approach outperforms existing methods with a remarkable margin. Our code is available at https://github.com/ChrisAllenMing/GPA-detection.
Despite the great progress made by deep CNNs in image semantic segmentation, they typically require a large number of densely-annotated images for training and are difficult to generalize to unseen object categories. Few-shot segmentation has thus been developed to learn to perform segmentation from only a few annotated examples. In this paper, we tackle the challenging few-shot segmentation problem from a metric learning perspective and present PANet, a novel prototype alignment network to better utilize the information of the support set. Our PANet learns class-specific prototype representat…
Tsinghua University（清华大学）；Tencent YouTu Lab（腾讯优图实验室）；University College London（伦敦大学学院）
Few-shot semantic segmentation aims to segment novel-class objects in a given query image with only a few labeled support images. Most advanced solutions exploit a metric learning framework that performs segmentation through matching each query feature to a learned class-specific prototype. However, this framework suffers from biased classification due to incomplete feature comparisons. To address this issue, we present an adaptive prototype representation by introducing class-specific and class-agnostic prototypes and thus construct complete sample pairs for learning semantic alignment with q…
Semi-supervised learning methods have been explored in medical image segmentation tasks due to the scarcity of pixel-level annotation in the real scenario. Proto-type alignment based consistency constraint is an intuitional and plausible solu-tion to explore the useful information in the unlabeled data. In this paper, we propose a mutual- and self- prototype alignment (MSPA) framework to better utilize the unlabeled data. In specific, mutual-prototype alignment enhances the information interaction between labeled and unlabeled data. The mutual-prototype alignment imposes two consistency constr…
Prototypical Progressive Alignment and Reweighting for Generalizable Semantic Segmentation
School of Computer Science and Cyber Engineering, Guangzhou University（计算机科学与网络安全学院，广州大学）；Guangdong Key Laboratory of Intelligent Information Processing, College of Electronics and Information Engineering, Shenzhen University（智能信息处理重点实验室，电子与信息工程学院，深圳大学）；School of Electronics and Communication Engineering, Guangzhou University（电子与通信工程学院，广州大学）；Univ Rennes, INSA Rennes, CNRS, IETR - UMR 6164（里昂大学，INSA里昂，CNRS，IETR - UMR 6164）；Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen Key Laboratory of Advanced Machine Learning and Applications, Institute of Artificial Intelligence and Advanced Communication, College of Electronics and Information Engineering, Shenzhen University（智能信息处理重点实验室，先进机器学习与应用重点实验室，人工智能与高级通信研究所，电子与信息工程学院，深圳大学）；College of Mathematics and Statistics, Shenzhen University（数学与统计学院，深圳大学）
Tencent Youtu Lab（腾讯优图实验室）；The Chinese University of Hong Kong（香港中文大学）
Unsupervised Domain Adaptation (UDA) aims to adapt the model trained on the labeled source domain to an unlabeled target domain. In this paper, we present Prototypical Contrast Adaptation (ProCA), a simple and efficient contrastive learning method for unsupervised domain adaptive semantic segmentation. Previous domain adaptation methods merely consider the alignment of the intra-class representational distributions across various domains, while the inter-class structural relationship is insufficiently explored, resulting in the aligned representations on the target domain might not be as easil…
DTFormer: Text-Guided Semantic Alignment for RGB-D Segmentation
Southeast University（东南大学）；Technical University of Munich（慕尼黑工业大学）；City University of Macau（澳门城市大学）；University of Science and Technology of China（中国科学技术大学）
School of Medical Imaging, Xuzhou Medical University（徐州医科大学医学影像学院）；The Hong Kong Polytechnic University（香港理工大学）
With the widespread success of deep learning in biomedical image segmentation, domain shift becomes a critical and challenging problem, as the gap between two domains can severely affect model performance when deployed to unseen data with heterogeneous features. To alleviate this problem, we present a novel unsupervised domain adaptation network, for generalizing models learned from the labeled source domain to the unlabeled target domain for cross-modality biomedical image segmentation. Specifically, our approach consists of two key modules, a conditional domain discriminator~(CDD) and a cate…
Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
Department of Computer Science and Artificial Intelligence, DaSCI Institute, University of Granada, Granada, Spain（计算机科学与人工智能系，DaSCI研究所，格拉纳达大学，格拉纳达，西班牙）；Department of Mathematics ``Tullio Levi-Civita'', University of Padova, Padova, Italy（托里利-西维塔数学系，帕多瓦大学，帕多瓦，意大利）
ShanghaiTech University（上海科技大学）；University of Chinese Academy of Sciences（中国科学院大学）
Zero-shot learning (ZSL) aims to recognize objects of novel classes without any training samples of specific classes, which is achieved by exploiting the semantic information and auxiliary datasets. Recently most ZSL approaches focus on learning visual-semantic embeddings to transfer knowledge from the auxiliary datasets to the novel classes. However, few works study whether the semantic information is discriminative or not for the recognition task. To tackle such problem, we propose a coupled dictionary learning approach to align the visual-semantic structures using the class prototypes, wher…
Zero-Shot Skeleton-Based Action Recognition With Prototype-Guided Feature Alignment
School of Software Engineering, South China University of Technology（南方科技大学软件工程学院）；South China University of Technology（南方科技大学）；Pazhou Lab（琶洲实验室）；School of Future Technology, South China University of Technology（未来技术学院）；Peng Cheng Laboratory（鹏城实验室）；Key Laboratory of Big Data and Intelligent Robot (South China University of Technology), Ministry of Education（大数据与智能机器人重点实验室）
The success of existing multi-view clustering relies on the assumption of sample integrity across multiple views. However, in real-world scenarios, samples of multi-view are partially available due to data corruption or sensor failure, which leads to incomplete multi-view clustering study (IMVC). Although several attempts have been proposed to address IMVC, they suffer from the following drawbacks: i) Existing methods mainly adopt cross-view contrastive learning forcing the representations of each sample across views to be exactly the same, which might ignore view discrepancy and flexibility i…
National University of Singapore（新加坡国立大学）；National Laboratory of Pattern Recognition (NLPR), Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所模式识别国家重点实验室）；Nankai University（南开大学）；Meitu Inc.（美图公司）
In this work, we attempt to address unsupervised domain adaptation by devising simple and compact conditional domain adversarial training methods. We first revisit the simple concatenation conditioning strategy where features are concatenated with output predictions as the input of the discriminator. We find the concatenation strategy suffers from the weak conditioning strength. We further demonstrate that enlarging the norm of concatenated predictions can effectively energize the conditional domain alignment. Thus we improve concatenation conditioning by normalizing the output predictions to …
Diverse Normal Prototypes-Guided Contrastive Reconstruction for Medical Anomaly Detection
School of Mathematics, Shandong University（山东大学数学学院）；Department of Interventional Medicine and Minimally Invasive Oncology, The Second Qilu Hospital of Shandong University（山东大学第二齐鲁医院介入医学与微创肿瘤科）；Department of Neurosurgery, The Second Qilu Hospital of Shandong University（山东大学第二齐鲁医院神经外科）
Indian Institute of Science（印度科学学院）；Google Research（谷歌研究院）
Universal Domain Adaptation (UniDA) deals with the problem of knowledge transfer between two datasets with domain-shift as well as category-shift. The goal is to categorize unlabeled target samples, either into one of the "known" categories or into a single "unknown" category. A major problem in UniDA is negative transfer, i.e. misalignment of "known" and "unknown" classes. To this end, we first uncover an intriguing tradeoff between negative-transfer-risk and domain-invariance exhibited at different layers of a deep network. It turns out we can strike a balance between these two metrics at a …
Peking University（北京大学）；South China University of Technology（华南理工大学）；National University of Singapore（新加坡国立大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
This paper studies a new, practical but challenging problem, called Class-Incremental Unsupervised Domain Adaptation (CI-UDA), where the labeled source domain contains all classes, but the classes in the unlabeled target domain increase sequentially. This problem is challenging due to two difficulties. First, source and target label sets are inconsistent at each time step, which makes it difficult to conduct accurate domain alignment. Second, previous target classes are unavailable in the current step, resulting in the forgetting of previous knowledge. To address this problem, we propose a nov…
Decouple, Purify and Unite: Semantic-Structural Prototype Learning for Federated Medical Segmentation
Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（北京协和医院，中国医学科学院北京协和医学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Nanyang Technological University（南洋理工大学）；Ant Group（蚂蚁集团）；Shanghai Innovation Institute（上海创新研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Fourth Military Medical University（中国人民解放军空军军医大学）；Macao Polytechnic University（澳门理工大学）；Wuhan University（武汉大学）；The Hong Kong Polytechnic University（香港理工大学）
PRIOR: Prototype Representation Joint Learning from Medical Images and Reports
Southern University of Science and Technology（南方科技大学）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）；The University of Hong Kong（香港大学）；University of British Columbia（不列颠哥伦比亚大学）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）
Temporal Prototyping and Hierarchical Alignment for Unsupervised Video-based Visible-Infrared Person Re-Identification
College of Control Science and Engineering, Zhejiang University（控制科学与工程学院，浙江大学）；School of Computer Science and Technology, Zhejiang University of Water Resources and Electric Power（计算机科学与技术学院，浙江水利电力大学）
University of Kentucky（肯塔基大学）；Eastern Kentucky University（东肯塔基大学）
Most research on domain adaptation has focused on the purely unsupervised setting, where no labeled examples in the target domain are available. However, in many real-world scenarios, a small amount of labeled target data is available and can be used to improve adaptation. We address this semi-supervised setting and propose to use dynamic feature alignment to address both inter- and intra-domain discrepancy. Unlike previous approaches, which attempt to align source and target features within a mini-batch, we propose to align the target features to a set of dynamically updated class prototypes,…
ERANet: Edge Replacement Augmentation for Semi-Supervised Meniscus Segmentation with Prototype Consistency Alignment and Conditional Self-Training
The Chinese University of Hong Kong（香港中文大学）；University of Edinburgh（爱丁堡大学）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；The Chinese University of Hong Kong Medical Centre（香港中文大学医学中心）；Southeast University（东南大学）；Xi'an Jiaotong Liverpool University（西交利物浦大学）；Nanchang University（南昌大学）
BoRAD: Bootstrap your Own Representations for Multi-class Anomaly Detection
Department of Artificial Intelligence, FPT University（FPT大学人工智能系）；Department of IT, FPT University（FPT大学信息技术系）；Department of Computing Fundamental, FPT University（FPT大学计算基础系）
PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology
Friedrich-Alexander-Universität（弗里德里希-亚历山大-埃尔朗根-纽伦堡大学）；Klinikum Nürnberg, Paracelsus Medical University（帕拉塞尔苏斯医科大学纽伦堡医院）；Ludwig-Maximilians-Universität München（慕尼黑大学）；National Institute of Mental Health and Neurosciences (NIMHANS)（国家精神卫生与神经科学研究院）
Peking University（北京大学）；University of Oxford（牛津大学）；JD Explore Academy（京东探索研究院）；Wuhan University（武汉大学）；Peng Cheng Laboratory（鹏城实验室）
Few-shot class-incremental learning (FSCIL) has been a challenging problem as only a few training samples are accessible for each novel class in the new sessions. Finetuning the backbone or adjusting the classifier prototypes trained in the prior sessions would inevitably cause a misalignment between the feature and classifier of old classes, which explains the well-known catastrophic forgetting problem. In this paper, we deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse, which reveals that the last-layer features of the same clas…
GRASP: Granularity-Aware Region Alignment and Semantic Prototype Learning for Fine-Grained Cross-Modal Understanding in Drone Views
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Tencent Data Platform（腾讯数据平台）；University of Science and Technology of China（中国科学技术大学）
Generalized Zero-Shot Learning (GZSL) aims to recognize new categories with auxiliary semantic information,e.g., category attributes. In this paper, we handle the critical issue of domain shift problem, i.e., confusion between seen and unseen categories, by progressively improving cross-domain transferability and category discriminability of visual representations. Our approach, named Dual Progressive Prototype Network (DPPN), constructs two types of prototypes that record prototypical visual patterns for attributes and categories, respectively. With attribute prototypes, DPPN alternately sear…
South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（教育部大数据与智能机器人重点实验室）；National University of Singapore（新加坡国立大学）
We study a practical domain adaptation task, called source-free unsupervised domain adaptation (UDA) problem, in which we cannot access source domain data due to data privacy issues but only a pre-trained source model and unlabeled target data are available. This task, however, is very difficult due to one key challenge: the lack of source data and target domain labels makes model adaptation very challenging. To address this, we propose to mine the hidden knowledge in the source model and exploit it to generate source avatar prototypes (i.e., representative features for each source class) as w…
