Image generating neural networks are mostly viewed as black boxes, where any change in the input can have a number of globally effective changes on the output. In this work, we propose a method for learning disentangled representations to allow for localized image manipulations. We use face images as our example of choice. Depending on the image region, identity and other facial attributes can be modified. The proposed network can transfer parts of a face such as shape and color of eyes, hair, mouth, etc.~directly between persons while all other parts of the face remain unchanged. The network allows to generate modified images which appear like realistic images. Our model learns disentangled representations by weak supervision. We propose a localized resnet autoencoder optimized using several loss functions including a loss based on the semantic segmentation, which we interpret as masks, and a loss which enforces disentanglement by decomposition of the latent space into statistically independent subspaces. We evaluate the proposed solution w.r.t. disentanglement and generated image quality. Convincing results are demonstrated using the CelebA dataset.
Semi-Supervised Gaze Estimation via Disentangled Subspace Contrastive Learning
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, China（合成视觉基础科学国家重点实验室，四川大学，成都，中国）；College of Computer Science, Sichuan University, Chengdu, China（计算机学院，四川大学，成都，中国）
Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
Technische Universität Berlin (TU Berlin)（柏林工业大学）；Max Planck School of Cognition（马克斯·普朗克认知学学院）；Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)（康拉德·楚泽学习与智能系统卓越学院）；BASF SE（巴斯夫公司）；Korea University（高丽大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；BIFOLD—Berlin Institute for the Foundations of Learning and Data（BIFOLD——柏林学习与数据基础研究所）；Freie Universität Berlin (FU Berlin)（柏林自由大学）
Although Generative Adversarial Networks (GANs) have made significant progress in face synthesis, there lacks enough understanding of what GANs have learned in the latent representation to map a random code to a photo-realistic image. In this work, we propose a framework called InterFaceGAN to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space. We first find that GANs learn various semantics in some linear subspaces of the latent space. After identifying these subspaces, we can r…
Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
Macquarie University（麦考瑞大学）；York University（约克大学）；Northern Illinois University（北伊利诺伊大学）；University of Technology Sydney（悉尼科技大学）；North South University（北南大学）；Lancaster University（兰卡斯特大学）；Australian National University（澳大利亚国立大学）
Generative models that learn disentangled representations for different factors of variation in an image can be very useful for targeted data augmentation. By sampling from the disentangled latent subspace of interest, we can efficiently generate new data necessary for a particular task. Learning disentangled representations is a challenging problem, especially when certain factors of variation are difficult to label. In this paper, we introduce a novel architecture that disentangles the latent space into two complementary subspaces by using only weak supervision in form of pairwise similarity…
Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization
Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；Department of Health Technology & Informatics, The Hong Kong Polytechnic University（香港理工大学健康科技与信息学系）；Department of Statistics and Actuarial Science, The University of Hong Kong（香港大学统计与精算科学系）；Department of Clinical Neurosciences and Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学临床神经科学系和应用数学与理论物理系；邓迪大学科学与工程学院和医学院）；School of Science and Engineering and School of Medicine, University of Dundee, UK
Learning Group Actions In Disentangled Latent Image Representations
Genuity Systems Limited（盖尼系统有限公司）；Computer Science（计算机科学）；Electrical and Computer Engineering（电气与计算机工程）；School of Engineering and Applied Science, University of Virginia（工程与应用科学学院，弗吉尼亚大学）
Fair representation learning aims to encode invariant representation with respect to the protected attribute, such as gender or age. In this paper, we design Fairness-aware Disentangling Variational AutoEncoder (FD-VAE) for fair representation learning. This network disentangles latent space into three subspaces with a decorrelation loss that encourages each subspace to contain independent information: 1) target attribute information, 2) protected attribute information, 3) mutual attribute information. After the representation learning, this disentangled representation is leveraged for fairer …
Cross-Camera Cow Identification via Disentangled Representation Learning
College of Information Science and Engineering, Shandong Agricultural University（信息科学与工程学院，山东农业大学）；Centre for Vision Speech and Signal Processing (CVSSP), University of Surrey（视觉语音与信号处理中心（CVSSP）， Surrey 大学）；Australian Institute for Machine Learning, University of Adelaide（机器学习研究院，阿德莱德大学）
A disentangled representation encodes information about the salient factors of variation in the data independently. Although it is often argued that this representational format is useful in learning to solve many real-world down-stream tasks, there is little empirical evidence that supports this claim. In this paper, we conduct a large-scale study that investigates whether disentangled representations are more suitable for abstract reasoning tasks. Using two new tasks similar to Raven's Progressive Matrices, we evaluate the usefulness of the representations learned by 360 state-of-the-art uns…
Guangdong University of Technology（广东工业大学）；Nanyang Technological University（南洋理工大学）；Carnegie Mellon University（卡内基梅隆大学）；Foshan University（佛山科学技术学院）
Domain adaptation is an important but challenging task. Most of the existing domain adaptation methods struggle to extract the domain-invariant representation on the feature space with entangling domain information and semantic information. Different from previous efforts on the entangled feature space, we aim to extract the domain invariant semantic information in the latent disentangled semantic representation (DSR) of the data. In DSR, we assume the data generation process is controlled by two independent sets of variables, i.e., the semantic latent variables and the domain latent variables…
Positive Semi-definite Latent Factor Grouping-Boosted Cluster-reasoning Instance Disentangled Learning for WSI Representation
Fu Foundation School of Engineering and Applied Science, Columbia University（哥伦比亚大学工程与应用科学学院）；Signal Processing Laboratory (LT55), École Polytechnique Fédérale de Lausanne (EPFL)（瑞士联邦理工学院信号处理实验室）；Radiology Department, Centre Hospitalier Universitaire Vaudois（瓦乌多大学医院放射科）；Jingfeng Laboratory, Chongqing, China（重庆 Jingfeng 实验室）；Centre for Smart Health, School of Nursing, The Hong Kong Polytechnic University（香港理工大学智能健康中心）
DiscoVL: Unveiling Disentangled C ross-Modal Representation Learning via Orthogonal Adversarial Regularization for V ision-Language Models
Shandong Artificial Intelligence Institute（山东人工智能研究院）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；University of Florida（佛罗里达大学）
CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation
DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
Face presentation attack detection (PAD) has been an urgent problem to be solved in the face recognition systems. Conventional approaches usually assume the testing and training are within the same domain; as a result, they may not generalize well into unseen scenarios because the representations learned for PAD may overfit to the subjects in the training set. In light of this, we propose an efficient disentangled representation learning for cross-domain face PAD. Our approach consists of disentangled representation learning (DR-Net) and multi-domain learning (MD-Net). DR-Net learns a pair of …
Learning disentangled representations from visual data, where different high-level generative factors are independently encoded, is of importance for many computer vision tasks. Solving this problem, however, typically requires to explicitly label all the factors of interest in training images. To alleviate the annotation cost, we introduce a learning setting which we refer to as "reference-based disentangling". Given a pool of unlabeled images, the goal is to learn a representation where a set of target factors are disentangled from others. The only supervision comes from an auxiliary "refere…
Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS)
Department of Electrical & Computer Engineering, University of Delaware, USA（电气与计算机工程系，德雷克塞尔大学）；Naval Surface Warfare Center, Panama City Division, Panama City, Florida, USA（Panama City 分部，海军水面 warfare 中心）；Department of Navy award N00014-24-1-2259 issued by the Office of Naval Research（海军部奖项 N00014-24-1-2259，由海军研究办公室颁发）；Department of Computer & Information Sciences, University of Delaware, USA（计算机与信息科学系，德雷克塞尔大学）
Xi’an Jiaotong University（西安交通大学）；Microsoft Research Asia（微软亚洲研究院）
Obtaining the human-like perception ability of abstracting visual concepts from concrete pixels has always been a fundamental and important target in machine learning research fields such as disentangled representation learning and scene decomposition. Towards this goal, we propose an unsupervised transformer-based Visual Concepts Tokenization framework, dubbed VCT, to perceive an image into a set of disentangled visual concept tokens, with each concept token responding to one type of independent visual concept. Particularly, to obtain these concept tokens, we only use cross-attention to extra…
We focus on explicitly learning disentangled representation for natural image generation, where the underlying spatial structure and the rendering on the structure can be independently controlled respectively, yet using no tuple supervision. The setting is significant since tuple supervision is costly and sometimes even unavailable. However, the task is highly unconstrained and thus ill-posed. To address this problem, we propose to introduce an auxiliary domain which shares a common underlying-structure space with the target domain, and we make a partially shared latent space assumption. The k…
The semantically disentangled latent subspace in GAN provides rich interpretable controls in image generation. This paper includes two contributions on semantic latent subspace analysis in the scenario of face generation using StyleGAN2. First, we propose a novel approach to disentangle latent subspace semantics by exploiting existing face analysis models, e.g., face parsers and face landmark detectors. These models provide the flexibility to construct various criterions with very concrete and interpretable semantic meanings (e.g., change face shape or change skin color) to restrict latent sub…
We present a framework for learning disentangled representation of CapsNet by information bottleneck constraint that distills information into a compact form and motivates to learn an interpretable factorized capsule. In our $β$-CapsNet framework, hyperparameter $β$ is utilized to trade-off disentanglement and other tasks, variational inference is utilized to convert the information bottleneck term into a KL divergence that is approximated as a constraint on the mean of the capsule. For supervised learning, class independent mask vector is used for understanding the types of variations synthet…
Sensory data are often comprised of independent content and transformation factors. For example, face images may have shapes as content and poses as transformation. To infer separately these factors from given data, various ``disentangling'' models have been proposed. However, many of these are supervised or semi-supervised, either requiring attribute labels that are often unavailable or disallowing for generalization over new contents. In this study, we introduce a novel deep generative model, called group-based variational autoencoders. In this, we assume no explicit labels, but a weaker for…
Building facial analysis systems that generalize to extreme variations in lighting and facial expressions is a challenging problem that can potentially be alleviated using natural-looking synthetic data. Towards that, we propose LEGAN, a novel synthesis framework that leverages perceptual quality judgments for jointly manipulating lighting and expressions in face images, without requiring paired training data. LEGAN disentangles the lighting and expression subspaces and performs transformations in the feature space before upscaling to the desired output image. The fidelity of the synthetic ima…
Parts of Speech-Grounded Subspaces in Vision-Language Models
Queen Mary University of London（伦敦玛丽女王大学）；National and Kapodistrian University of Athens（雅典国立卡波季斯特里安大学）；Archimedes/Athena RC（阿基米德/雅典研究与创新中心）；The Cyprus Institute（塞浦路斯研究所）
We present a framework to translate between 2D image views and 3D object shapes. Recent progress in deep learning enabled us to learn structure-aware representations from a scene. However, the existing literature assumes that pairs of images and 3D shapes are available for training in full supervision. In this paper, we propose SIST, a Self-supervised Image to Shape Translation framework that fulfills three tasks: (i) reconstructing the 3D shape from a single image; (ii) learning disentangled representations for shape, appearance and viewpoint; and (iii) generating a realistic RGB image from t…
MIT Interdisciplinary Center, Herzliya（赫兹利亚跨学科中心（MIT））；Stanford University（斯坦福大学）
Learning 3D generative models from a dataset of monocular images enables self-supervised 3D reasoning and controllable synthesis. State-of-the-art 3D generative models are GANs which use neural 3D volumetric representations for synthesis. Images are synthesized by rendering the volumes from a given camera. These models can disentangle the 3D scene from the camera viewpoint in any generated image. However, most models do not disentangle other factors of image formation, such as geometry and appearance. In this paper, we design a 3D GAN which can learn a disentangled model of objects, just from …
Generating videos predicting the future of a given sequence has been an area of active research in recent years. However, an essential problem remains unsolved: most of the methods require large computational cost and memory usage for training. In this paper, we propose a novel method for generating future prediction videos with less memory usage than the conventional methods. This is a critical stepping stone in the path towards generating videos with high image quality, similar to that of generated images in the latest works in the field of image generation. We achieve high-efficiency by tra…
Cornell University（康奈尔大学）；Uber AI Labs（Uber AI 实验室）；Sloan Kettering Institute（斯隆凯特琳研究所）
What makes images similar? To measure the similarity between images, they are typically embedded in a feature-vector space, in which their distance preserve the relative dissimilarity. However, when learning such similarity embeddings the simplifying assumption is commonly made that images are only compared to one unique measure of similarity. A main reason for this is that contradicting notions of similarities cannot be captured in a single space. To address this shortcoming, we propose Conditional Similarity Networks (CSNs) that learn embeddings differentiated into semantically distinct subs…
Disentangling factors of variation has become a very challenging problem on representation learning. Existing algorithms suffer from many limitations, such as unpredictable disentangling factors, poor quality of generated images from encodings, lack of identity information, etc. In this paper, we propose a supervised learning model called DNA-GAN which tries to disentangle different factors or attributes of images. The latent representations of images are DNA-like, in which each individual piece (of the encoding) represents an independent factor of the variation. By annihilating the recessive …
Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel our image formation process into three conditionally independent factors---shape, viewpoint, and texture---and present an end-to-end adversarial learning framework that join…
RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding
Technical University of Munich (TUM)（慕尼黑工业大学（TUM））；TUM University Hospital（慕尼黑工业大学医院）；Technical University of Munich, School of Medicine and Health（慕尼黑工业大学医学与健康学院）；Klinikum rechts der Isar（右伊萨尔医院）；Charité – Universitätsmedizin Berlin（柏林夏里特医学院）；Freie Universität Berlin（柏林自由大学）；Humboldt Universität zu Berlin（柏林洪堡大学）；Imperial College London（伦敦帝国理工学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；University Hospital Essen (AöR)（埃森大学医院（AöR））；Institute for Artificial Intelligence in Medicine (IKIM)（医学人工智能研究所（IKIM））；Institute of Interventional and Diagnostic Radiology and Neuroradiology（介入与诊断放射学及神经放射学研究所）；National Center for Tumor Diseases West（西部肿瘤疾病国家中心）
Nankai University（南开大学）；Cardiff University（卡迪夫大学）；Chinese Academy of Sciences（中国科学院）
In this paper we propose a unified framework to simultaneously discover the number of clusters and group the data points into them using subspace clustering. Real data distributed in a high-dimensional space can be disentangled into a union of low-dimensional subspaces, which can benefit various applications. To explore such intrinsic structure, state-of-the-art subspace clustering approaches often optimize a self-representation problem among all samples, to construct a pairwise affinity graph for spectral clustering. However, a graph with pairwise similarities lacks robustness for segmentatio…
This paper explores a comparative study of both the linear and kernel implementations of three of the most popular Appearance-based Face Recognition projection classes, these being the methodologies of Principal Component Analysis, Linear Discriminant Analysis and Independent Component Analysis. The experimental procedure provides a platform of equal working conditions and examines the ten algorithms in the categories of expression, illumination, occlusion and temporal delay. The results are then evaluated based on a sequential combination of assessment tools that facilitate both intuitive and…
Facial makeup transfer aims to render a non-makeup face image in an arbitrary given makeup one while preserving face identity. The most advanced method separates makeup style information from face images to realize makeup transfer. However, makeup style includes several semantic clear local styles which are still entangled together. In this paper, we propose a novel unified adversarial disentangling network to further decompose face images into four independent components, i.e., personal identity, lips makeup style, eyes makeup style and face makeup style. Owing to the further disentangling of…
Object Transfiguration replaces an object in an image with another object from a second image. For example it can perform tasks like "putting exactly those eyeglasses from image A on the nose of the person in image B". Usage of exemplar images allows more precise specification of desired modifications and improves the diversity of conditional image generation. However, previous methods that rely on feature space operations, require paired data and/or appearance models for training or disentangling objects from background. In this work, we propose a model that can learn object transfiguration f…
IT University of Copenhagen（哥本哈根信息技术大学）；Leibniz Universität Hannover（汉诺威莱布尼茨大学）
We propose a general, prior-free approach for the uncalibrated non-rigid structure-from-motion problem for modelling and analysis of non-rigid objects such as human faces. The word general refers to an approach that recovers the non-rigid affine structure and motion from 2D point correspondences by assuming that (1) the non-rigid shapes are generated by a linear combination of rigid 3D basis shapes, (2) that the non-rigid shapes are affine in nature, i.e., they can be modelled as deviations from the mean, rigid shape, (3) and that the basis shapes are statistically independent. In contrast to …
Adaptive Disentangled Representation Learning for Incomplete Multi-View Multi-Label Classification
College of Science, National University of Defense Technology（国防科技大学科学学院）；College of Artificial Intelligence, Harbin Institute of Technology（哈尔滨工业大学人工智能学院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）
