Deep learning models exhibit limited generalizability across different domains. Specifically, transferring knowledge from available entangled domain features(source/target domain) and categorical features to new unseen categorical features in a target domain is an interesting and difficult problem that is rarely discussed in the current literature. This problem is essential for many real-world applications such as improving diagnostic classification or prediction in medical imaging. To address this problem, we propose Mutual-Information-based Disentangled Neural Networks (MIDNet) to extract generalizable features that enable transferring knowledge to unseen categorical features in target domains. The proposed MIDNet is developed as a semi-supervised learning paradigm to alleviate the dependency on labeled data. This is important for practical applications where data annotation requires rare expertise as well as intense time and labor. We demonstrate our method on handwritten digits datasets and a fetal ultrasound dataset for image classification tasks. Experiments show that our method outperforms the state-of-the-art and achieve expected performance with sparsely labeled data.
University of Electronic Science and Technology of China（电子科技大学）；Shanghai AI Lab（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；ShanghaiTech University（上海科技大学）；Sensetime Research（商汤科技研究院）；Sichuan University（四川大学）；Sichuan Cancer Hospital and Institute（四川省肿瘤医院（四川省肿瘤研究所））
Generalization to previously unseen images with potential domain shifts and different styles is essential for clinically applicable medical image segmentation, and the ability to disentangle domain-specific and domain-invariant features is key for achieving Domain Generalization (DG). However, existing DG methods can hardly achieve effective disentanglement to get high generalizability. To deal with this problem, we propose an efficient Contrastive Domain Disentanglement and Style Augmentation (CDDSA) framework for generalizable medical image segmentation. First, a disentangle network is propo…
University of Electronic Science and Technology of China（电子科技大学）；Shanghai AI Lab（上海人工智能实验室）；ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）
Efficiently utilizing discriminative features is crucial for convolutional neural networks to achieve remarkable performance in medical image segmentation and is also important for model generalization across multiple domains, where letting model recognize domain-specific and domain-invariant information among multi-site datasets is a reasonable strategy for domain generalization. Unfortunately, most of the recent disentangle networks are not directly adaptable to unseen-domain datasets because of the limitations of offered data distribution. To tackle this deficiency, we propose Contrastive D…
For Domain Generalizable Object Detection (DGOD), Disentangled Representation Learning (DRL) helps a lot by explicitly disentangling Domain-Invariant Representations (DIR) from Domain-Specific Representations (DSR). Considering the domain category is an attribute of input data, it should be feasible for networks to fit a specific mapping which projects DSR into feature channels exclusive to domain-specific information, and thus much cleaner disentanglement of DIR from DSR can be achieved simply on channel dimension. Inspired by this idea, we propose a novel DRL method for DGOD, which is termed…
Limited amount of labelled training data are a common problem in medical imaging. This makes it difficult to train a well-generalised model and therefore often leads to failure in unknown domains. Hippocampus segmentation from magnetic resonance imaging (MRI) scans is critical for the diagnosis and treatment of neuropsychatric disorders. Domain differences in contrast or shape can significantly affect segmentation. We address this issue by disentangling a T1-weighted MRI image into its content and domain. This separation enables us to perform a domain transfer and thus convert data from new so…
Decoupling Disease, Covariates, and Individual Variability: A Unified Disentanglement Framework for Medical Image Classification
Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；Ruijin Hospital（瑞金医院）；Fudan University（复旦大学）；Shandong Provincial Hospital Affiliated to Shandong First Medical University（山东第一医科大学附属省立医院）；Air Force Medical University（空军军医大学）
Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation
Shanghai Jiao Tong University（上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；University of Science and Technology of China（中国科学技术大学）；The University of Hong Kong（香港大学）
University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）；University of Pittsburgh（匹兹堡大学）；Brain and Mind Centre, University of Sydney（悉尼大学脑与心智中心）
Recent advances in unsupervised domain adaptation (UDA) techniques have witnessed great success in cross-domain computer vision tasks, enhancing the generalization ability of data-driven deep learning architectures by bridging the domain distribution gaps. For the UDA-based cross-domain object detection methods, the majority of them alleviate the domain bias by inducing the domain-invariant feature generation via adversarial learning strategy. However, their domain discriminators have limited classification ability due to the unstable adversarial training process. Therefore, the extracted feat…
National Taiwan University（台湾大学）；MOST Joint Research Center for AI Technology and All Vista Healthcare（科技部AI技术与全视医疗照护联合研究中心）；ASUS Intelligent Cloud Services（华硕智能云服务）
Person re-identification (re-ID) aims at recognizing the same person from images taken across different cameras. To address this challenging task, existing re-ID models typically rely on a large amount of labeled training data, which is not practical for real-world applications. To alleviate this limitation, researchers now targets at cross-dataset re-ID which focuses on generalizing the discriminative ability to the unlabeled target domain when given a labeled source domain dataset. To achieve this goal, our proposed Pose Disentanglement and Adaptation Network (PDA-Net) aims at learning deep …
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
Unsupervised domain adaptive (UDA) person re-identification (ReID) aims to transfer the knowledge from the labeled source domain to the unlabeled target domain for person matching. One challenge is how to generate target domain samples with reliable labels for training. To address this problem, we propose a Disentanglement-based Cross-Domain Feature Augmentation (DCDFA) strategy, where the augmented features characterize well the target and source domain data distributions while inheriting reliable identity labels. Particularly, we disentangle each sample feature into a robust domain-invariant…
Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；MILA（蒙特利尔学习算法研究所）；Université de Montréal（蒙特利尔大学）
Deep image translation methods have recently shown excellent results, outputting high-quality images covering multiple modes of the data distribution. There has also been increased interest in disentangling the internal representations learned by deep methods to further improve their performance and achieve a finer control. In this paper, we bridge these two objectives and introduce the concept of cross-domain disentanglement. We aim to separate the internal representation into three parts. The shared part contains information for both domains. The exclusive parts, on the other hand, contain o…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Visible-infrared person re-identification (VI-ReID) is an important task in night-time surveillance applications, since visible cameras are difficult to capture valid appearance information under poor illumination conditions. Compared to traditional person re-identification that handles only the intra-modality discrepancy, VI-ReID suffers from additional cross-modality discrepancy caused by different types of imaging systems. To reduce both intra- and cross-modality discrepancies, we propose a Hierarchical Cross-Modality Disentanglement (Hi-CMD) method, which automatically disentangles ID-disc…
Disentangle-then-Align: Non-Iterative Hybrid Multimodal Image Registration via Cross-Scale Feature Disentanglement
Faculty of Engineering and IT, University of Technology Sydney（新南威尔士大学工程与信息技术学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
National Taiwan University（台湾大学）；Northwestern University（西北大学）；National Chiao Tung University（台湾交通大学）
While representation learning aims to derive interpretable features for describing visual data, representation disentanglement further results in such features so that particular image attributes can be identified and manipulated. However, one cannot easily address this task without observing ground truth annotation for the training data. To address this problem, we propose a novel deep learning model of Cross-Domain Representation Disentangler (CDRD). By observing fully annotated source-domain data and unlabeled target-domain data of interest, our model bridges the information across data dom…
Graph-guided Cross-composition Feature Disentanglement for Compositional Zero-shot Learning
PowerChina Huadong Engineering Corporation Limited（国家电力投资集团华东工程公司）；Hangzhou Dianzi University（杭州电子科技大学）；The University of Manchester（曼彻斯特大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Zhejiang University（浙江大学）
Carnegie Mellon University（卡内基梅隆大学）；Chinese Academy of Science（中国科学院）；University of Cambridge（剑桥大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Alan Turing Institute（艾伦·图灵研究所）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
A fundamental challenge for machine learning models is generalizing to out-of-distribution (OOD) data, in part due to spurious correlations. To tackle this challenge, we first formalize the OOD generalization problem as constrained optimization, called Disentanglement-constrained Domain Generalization (DDG). We relax this non-trivial constrained optimization problem to a tractable form with finite-dimensional parameterization and empirical approximation. Then a theoretical analysis of the extent to which the above transformations deviates from the original problem is provided. Based on the tra…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）；University of Oxford（牛津大学）
Existing fully-supervised person re-identification (ReID) methods usually suffer from poor generalization capability caused by domain gaps. The key to solving this problem lies in filtering out identity-irrelevant interference and learning domain-invariant person representations. In this paper, we aim to design a generalizable person ReID framework which trains a model on source domains yet is able to generalize/perform well on target domains. To achieve this goal, we propose a simple yet effective Style Normalization and Restitution (SNR) module. Specifically, we filter out style variations (…
Vision-Language Attribute Disentanglement and Reinforcement for Lifelong Person Re-Identification
Wangxuan Institute of Computer Technology, Peking University, Beijing, China（北京大学王轩计算机技术研究院，北京，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China（华中科技大学人工智能与自动化学院，武汉，中国）
Over the last few decades, many aspects of human life have been enhanced with virtual domains, from the advent of digital assistants such as Amazon's Alexa and Apple's Siri to the latest metaverse efforts of the rebranded Meta. These trends underscore the importance of generating photorealistic visual depictions of humans. This has led to the rapid growth of so-called deepfake and talking-head generation methods in recent years. Despite their impressive results and popularity, they usually lack certain qualitative aspects such as texture quality, lips synchronization, or resolution, and practi…
We present the task of differential face morph attack detection using a conditional generative network (cGAN). To determine whether a face image in an identification document, such as a passport, is morphed or not, we propose an algorithm that learns to implicitly disentangle identities from the morphed image conditioned on the trusted reference image using the cGAN. Furthermore, the proposed method can also recover some underlying information about the second subject used in generating the morph. We performed experiments on AMSL face morph, MorGAN, and EMorGAN datasets to demonstrate the effe…
Text-guided Feature Disentanglement for Cross-modal Gait Recognition
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University（福建城市智能感知与计算重点实验室，厦门大学）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，中华人民共和国教育部，厦门大学）
Peking University（北京大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Tsinghua University（清华大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
For years, researchers have been devoted to generalizable object perception and manipulation, where cross-category generalizability is highly desired yet underexplored. In this work, we propose to learn such cross-category skills via Generalizable and Actionable Parts (GAParts). By identifying and defining 9 GAPart classes (lids, handles, etc.) in 27 object categories, we construct a large-scale part-centric interactive dataset, GAPartNet, where we provide rich, part-level annotations (semantics, poses) for 8,489 part instances on 1,166 objects. Based on GAPartNet, we investigate three cross-c…
Deep learning models are vulnerable to adversarial examples and make incomprehensible mistakes, which puts a threat on their real-world deployment. Combined with the idea of adversarial training, preprocessing-based defenses are popular and convenient to use because of their task independence and good generalizability. Current defense methods, especially purification, tend to remove ``noise" by learning and recovering the natural images. However, different from random noise, the adversarial patterns are much easier to be overfitted during model training due to their strong correlation to the i…
Domain adaptation aims to mitigate the domain gap when transferring knowledge from an existing labeled domain to a new domain. However, existing disentanglement-based methods do not fully consider separation between domain-invariant and domain-specific features, which means the domain-invariant features are not discriminative. The reconstructed features are also not sufficiently used during training. In this paper, we propose a novel enhanced separable disentanglement (ESD) model. We first employ a disentangler to distill domain-invariant and domain-specific features. Then, we apply feature se…
Cross-domain disentanglement is the problem of learning representations partitioned into domain-invariant and domain-specific representations, which is a key to successful domain transfer or measuring semantic distance between two domains. Grounded in information theory, we cast the simultaneous learning of domain-invariant and domain-specific representations as a joint objective of multiple information constraints, which does not require adversarial training or gradient reversal layers. We derive a tractable bound of the objective and propose a generative model named Interaction Information A…
This paper presents a generic method for generating full facial 3D animation from speech. Existing approaches to audio-driven facial animation exhibit uncanny or static upper face animation, fail to produce accurate and plausible co-articulation or rely on person-specific models that limit their scalability. To improve upon existing models, we propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. At the core of our approach is a categorical latent space for facial animation that disentangles audio-correlated and au…
Xi’an Jiaotong University（西安交通大学）；HKUST（香港科技大学）；Microsoft Research Asia（微软亚洲研究院）；EIT（电子与信息技术研究所）
Disentangled representation learning is one of the major goals of deep learning, and is a key step for achieving explainable and generalizable models. A well-defined theoretical guarantee still lacks for the VAE-based unsupervised methods, which are a set of popular methods to achieve unsupervised disentanglement. The Group Theory based definition of representation disentanglement mathematically connects the data transformations to the representations using the formalism of group. In this paper, built on the group-based definition and inspired by the n-th dihedral group, we first propose a the…
For medical image segmentation, imagine if a model was only trained using MR images in source domain, how about its performance to directly segment CT images in target domain? This setting, namely generalizable cross-modality segmentation, owning its clinical potential, is much more challenging than other related settings, e.g., domain adaptation. To achieve this goal, we in this paper propose a novel dual-normalization model by leveraging the augmented source-similar and source-dissimilar images during our generalizable segmentation. To be specific, given a single source domain, aiming to sim…
Visible-infrared person re-identification (VI-ReID) is a challenging and essential task in night-time intelligent surveillance systems. Except for the intra-modality variance that RGB-RGB person re-identification mainly overcomes, VI-ReID suffers from additional inter-modality variance caused by the inherent heterogeneous gap. To solve the problem, we present a carefully designed dual Gaussian-based variational auto-encoder (DG-VAE), which disentangles an identity-discriminable and an identity-ambiguous cross-modality feature subspace, following a mixture-of-Gaussians (MoG) prior and a standar…
Images captured in the low-light condition suffer from low visibility and various imaging artifacts, e.g., real noise. Existing supervised enlightening algorithms require a large set of pixel-aligned training image pairs, which are hard to prepare in practice. Though weakly-supervised or unsupervised methods can alleviate such challenges without using paired training images, some real-world artifacts inevitably get falsely amplified because of the lack of corresponded supervision. In this paper, instead of using perfectly aligned images for training, we creatively employ the misaligned real-wo…
Learning Representation and Synergy Invariances: A Povable Framework for Generalized Multimodal Face Anti-Spoofing
Beihang University（北京航空航天大学）；Nanyang Technological University（南洋理工大学）；Westlake University（西湖大学）；Great Bay University（大亚湾大学）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shandong University of Science and Technology（山东科技大学）
Efficient medical image segmentation aims to provide accurate pixel-wise predictions for medical images with a lightweight implementation framework. However, lightweight frameworks generally fail to achieve superior performance and suffer from poor generalizable ability on cross-domain tasks. In this paper, we explore the generalizable knowledge distillation for the efficient segmentation of cross-domain medical images. Considering the domain gaps between different medical datasets, we propose the Model-Specific Alignment Networks (MSAN) to obtain the domain-invariant representations. Meanwhil…
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京国家信息科学与技术研究中心）
In this paper, we aim to learn a semantic radiance field from multiple scenes that is accurate, efficient and generalizable. While most existing NeRFs target at the tasks of neural scene rendering, image synthesis and multi-view reconstruction, there are a few attempts such as Semantic-NeRF that explore to learn high-level semantic understanding with the NeRF structure. However, Semantic-NeRF simultaneously learns color and semantic label from a single ray with multiple heads, where the single ray fails to provide rich semantic information. As a result, Semantic NeRF relies on positional encod…
Environment-Invariant Subspace Learning for Generalizable Deepfake Detection
Tianjin University of Technology（天津理工大学）；Shandong Artificial Intelligence Institute（山东人工智能研究院）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））
