Deep neural networks can form high-level hierarchical representations of input data. Various researchers have demonstrated that these representations can be used to enable a variety of useful applications. However, such representations are typically based on the statistics within the data, and may not conform with the semantic representation that may be necessitated by the application. Conditional models are typically used to overcome this challenge, but they require large annotated datasets which are difficult to come by and costly to create. In this paper, we show that semantically-aligned representations can be generated instead with the help of a physics based engine. This is accomplished by creating a synthetic dataset with decoupled attributes, learning an encoder for the synthetic dataset, and augmenting prescribed attributes from the synthetic domain with attributes from the real domain. It is shown that the proposed (SYNTH-VAE-GAN) method can construct a conditional predictive-generative model of human face attributes without relying on real data labels.
University of Oxford（牛津大学）；Microsoft Research（微软研究院）
We would like to learn a representation of the data which decomposes an observation into factors of variation which we can independently control. Specifically, we want to use minimal supervision to learn a latent representation that reflects the semantics behind a specific grouping of the data, where within a group the samples share a common factor of variation. For example, consider a collection of face images grouped by identity. We wish to anchor the semantics of the grouping into a relevant and disentangled representation that we can easily exploit. However, existing deep probabilistic mod…
We explore and analyze the latent style space of StyleGAN2, a state-of-the-art architecture for image generation, using models pretrained on several different datasets. We first show that StyleSpace, the space of channel-wise style parameters, is significantly more disentangled than the other intermediate latent spaces explored by previous works. Next, we describe a method for discovering a large collection of style channels, each of which is shown to control a distinct visual attribute in a highly localized and disentangled manner. Third, we propose a simple method for identifying style chann…
HKUST（香港科技大学）；Xi’an Jiaotong University（西安交通大学）；Microsoft Research Asia（微软亚洲研究院）；EIT（欧洲创新与技术学院）
From the intuitive notion of disentanglement, the image variations corresponding to different factors should be distinct from each other, and the disentangled representation should reflect those variations with separate dimensions. To discover the factors and learn disentangled representation, previous methods typically leverage an extra regularization term when learning to generate realistic images. However, the term usually results in a trade-off between disentanglement and generation quality. For the generative models pretrained without any disentanglement term, the generated images show se…
Queen Mary University of London（伦敦玛丽女王大学）；University of Athens（雅典大学）；Cyprus Institute（塞浦路斯研究所）
Recently, a multitude of methods for image-to-image translation have demonstrated impressive results on problems such as multi-domain or multi-attribute transfer. The vast majority of such works leverages the strengths of adversarial learning and deep convolutional autoencoders to achieve realistic results by well-capturing the target data distribution. Nevertheless, the most prominent representatives of this class of methods do not facilitate semantic structure in the latent space, and usually rely on binary domain labels for test-time transfer. This leads to rigid models, unable to capture t…
Disentangled representations via score-based variational autoencoders
Center for Neural Science, New York University（纽约大学神经科学中心）；Center for Computational Neuroscience, Flatiron Institute（Flatiron研究所计算神经科学中心）；Center for Data Science, New York University（纽约大学数据科学中心）
Computer Vision Lab ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）；KU Leuven（鲁汶大学）
Attribute image manipulation has been a very active topic since the introduction of Generative Adversarial Networks (GANs). Exploring the disentangled attribute space within a transformation is a very challenging task due to the multiple and mutually-inclusive nature of the facial images, where different labels (eyeglasses, hats, hair, identity, etc.) can co-exist at the same time. Several works address this issue either by exploiting the modality of each domain/attribute using a conditional random vector noise, or extracting the modality from an exemplary image. However, existing methods cann…
Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, IDG/McGovern Institute for Brain Research, Peking University（北京大学-清华生命科学中心，先进跨学科研究院，IDG/麦克戈文脑科学研究院，北京大学）；Center of Quantitative Biology, Peking University（北京大学定量生物学中心）；School of Psychological and Cognitive Sciences, Key Laboratory of Machine Perception (Ministry of Education), Peking University（心理与认知科学学院，机器感知重点实验室（教育部），北京大学）
Generative Adversarial Networks (GANs) are able to learn mappings between simple, relatively low-dimensional, random distributions and points on the manifold of realistic images in image-space. The semantics of this mapping, however, are typically entangled such that meaningful image properties cannot be controlled independently of one another. Conditional GANs (cGANs) provide a potential solution to this problem, allowing specific semantics to be enforced during training. This solution, however, depends on the availability of precise labels, which are sometimes difficult or near impossible to…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
The performance of $β$-Variational-Autoencoders ($β$-VAEs) and their variants on learning semantically meaningful, disentangled representations is unparalleled. On the other hand, there are theoretical arguments suggesting the impossibility of unsupervised disentanglement. In this work, we shed light on the inductive bias responsible for the success of VAE-based architectures. We show that in classical datasets the structure of variance, induced by the generating factors, is conveniently aligned with the latent directions fostered by the VAE objective. This builds the pivotal bias on which the…
Closed-Loop Unsupervised Representation Disentanglement with $β$-VAE Distillation and Diffusion Probabilistic Feedback
Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所、东部技术研究所）；Shanghai Jiao Tong University（上海交通大学）；Xi’an Jiaotong University（西安交通大学）
Traditional face editing methods often require a number of sophisticated and task specific algorithms to be applied one after the other --- a process that is tedious, fragile, and computationally intensive. In this paper, we propose an end-to-end generative adversarial network that infers a face-specific disentangled representation of intrinsic face properties, including shape (i.e. normals), albedo, and lighting, and an alpha matte. We show that this network can be trained on "in-the-wild" images by incorporating an in-network physically-based image formation module and appropriate loss funct…
Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs
Shanghai Jiao Tong University（上海交通大学）；Ningbo Institute of Digital Twin（宁波数字孪生研究院）；Eastern Institute of Technology（东方理工大学）；University of Science and Technology of China（中国科学技术大学）；Yokohama National University（横滨国立大学）
Disentangled representation is a powerful technique to tackle domain shift problem in medical image analysis in unsupervised domain adaptation setting.However, previous methods only focus on exacting domain-invariant feature and ignore whether exacted feature is meaningful for downstream tasks.We propose a novel framework, called semantic-guided disentangled representation (SGDR), an effective method to exact semantically meaningful feature for segmentation task to improve performance of cross modality medical image segmentation in unsupervised domain adaptation setting. To exact the meaningfu…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
Image-to-image translation models have shown remarkable ability on transferring images among different domains. Most of existing work follows the setting that the source domain and target domain keep the same at training and inference phases, which cannot be generalized to the scenarios for translating an image from an unseen domain to another unseen domain. In this work, we propose the Unsupervised Zero-Shot Image-to-image Translation (UZSIT) problem, which aims to learn a model that can translate samples from image domains that are not observed during training. Accordingly, we propose a fram…
The Chinese University of Hong Kong（香港中文大学）；SenseTime Group Limited（商汤集团有限公司）；Peking University（北京大学）
This paper proposes learning disentangled but complementary face features with minimal supervision by face identification. Specifically, we construct an identity Distilling and Dispelling Autoencoder (D2AE) framework that adversarially learns the identity-distilled features for identity verification and the identity-dispelled features to fool the verification system. Thanks to the design of two-stream cues, the learned disentangled features represent not only the identity or attribute but the complete input image. Comprehensive evaluations further demonstrate that the proposed features not onl…
Massachusetts Institute of Technology（麻省理工学院）；Harvard Medical School（哈佛医学院）
State-of-the-art methods in generative representation learning yield semantic disentanglement, but typically do not consider physical scene parameters, such as geometry, albedo, lighting, or camera. We posit that inverse rendering, a way to reverse the rendering process to recover scene parameters from an image, can also be used to learn physically disentangled representations of scenes without supervision. In this paper, we show the utility of inverse rendering in learning representations that yield improved accuracy on downstream clustering, linear classification, and segmentation tasks with…
IIIT-Delhi（印度理工学院德里分校）；Arizona State University（亚利桑那州立大学）
Deep generative models like variational autoencoders approximate the intrinsic geometry of high dimensional data manifolds by learning low-dimensional latent-space variables and an embedding function. The geometric properties of these latent spaces has been studied under the lens of Riemannian geometry; via analysis of the non-linearity of the generator function. In new developments, deep generative models have been used for learning semantically meaningful `disentangled' representations; that capture task relevant attributes while being invariant to other attributes. In this work, we explore …
MIT（麻省理工学院）；University of California, Berkeley（加州大学伯克利分校）；Technion—Israel Institute of Technology（以色列理工学院）
Meaningful uncertainty quantification in computer vision requires reasoning about semantic information -- say, the hair color of the person in a photo or the location of a car on the street. To this end, recent breakthroughs in generative modeling allow us to represent semantic information in disentangled latent spaces, but providing uncertainties on the semantic latent variables has remained challenging. In this work, we provide principled uncertainty intervals that are guaranteed to contain the true semantic factors for any underlying generative model. The method does the following: (1) it u…
City University of Hong Kong（香港城市大学）；Snap Inc.（Snap公司）；USC Institute for Creative Technologies（南加州大学创意技术研究所）；Microsoft Cloud AI（微软云人工智能部门）
Face image manipulation via three-dimensional guidance has been widely applied in various interactive scenarios due to its semantically-meaningful understanding and user-friendly controllability. However, existing 3D-morphable-model-based manipulation methods are not directly applicable to out-of-domain faces, such as non-photorealistic paintings, cartoon portraits, or even animals, mainly due to the formidable difficulties in building the model for each specific face domain. To overcome this challenge, we propose, as far as we know, the first method to manipulate faces in arbitrary domains us…
Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Imperial College London（伦敦帝国理工学院）；University of Oxford（牛津大学）；Technical University of Munich（慕尼黑工业大学）
For visual manipulation tasks, we aim to represent image content with semantically meaningful features. However, learning implicit representations from images often lacks interpretability, especially when attributes are intertwined. We focus on the challenging task of extracting disentangled 3D attributes only from 2D image data. Specifically, we focus on human appearance and learn implicit pose, shape and garment representations of dressed humans from RGB images. Our method learns an embedding with disentangled latent representations of these three image properties and enables meaningful re-a…
Semimage: HSV-Based Semantic Image Encoding for Disentangled Text Representation
AI Lab at Department of Computer Engineering（计算机工程系人工智能实验室）；AriooBarzan Engineering Team and Information Technology（AriooBarzan工程团队和信息技术）；Shiraz University of Technology（谢兹大学技术学院）
ConceptScope: Characterizing Dataset Bias via Disentangled Visual Concepts
KAIST AI（韩国科学技术院人工智能研究所）；Helmholtz Munich（慕尼黑海德堡研究所）；Institute of Computational Biology（计算生物学研究所）；Computational Health Center（计算健康中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations. Although deep neural networks have exhibited superior performance in various tasks, the interpretability is always the Achilles' heel of deep neural networks. At present, deep neural networks obtain high discrimination power at the cost of low interpretability of their black-box representations. We believe that high model interpretability may help people to break several bottlenecks of deep learning, e.g., learning from ver…
Unified and Semantically Grounded Domain Adaptation for Medical Image Segmentation
Department of Electrical and Computer Engineering, University of Washington（电气与计算机工程系，华盛顿大学）；Department of Bioengineering, University of Washington（生物工程系，华盛顿大学）；Department of Radiology, University of Washington（放射学系，华盛顿大学）；Department of Electrical and Computer Engineering and Paul G. Allen School of Computer Science and Engineering, University of Washington（电气与计算机工程系及保罗·G·阿伦计算机科学与工程学院，华盛顿大学）；Department of Radiology and Imaging Sciences, University of Utah（放射学与影像科学系，犹他大学）
The GENEA Challenge 2026: A Large-Scale Disentangled Evaluation of Speech-Driven Gesture Generation on the Seamless Interaction Dataset
KTH Royal Institute of Technology（瑞典皇家理工学院）；Bielefeld University（比勒费尔德大学）；National Library of Sweden（瑞典国家图书馆）；Electronics and Telecommunications Research Institute (ETRI)（电子通信研究院）；Motorica AB（Motorica AB公司）
A2V: A Semi-Supervised Domain Adaptation Framework for Brain Vessel Segmentation via Two-Phase Training Angiography-to-Venography Translation
EURECOM（欧洲通信学院）；University of Siena（锡耶纳大学）；Centre Hospitalier Universitaire de Nice（尼斯大学医院中心）；Centre for Medical Image Computing University College London（伦敦大学学院医学影像计算中心）；e-health Center Universitat Oberta de Catalunya（加泰罗尼亚开放大学电子健康中心）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共医疗救助机构）
