Diagonal Attention and Style-based GAN for Content-Style Disentanglement in Image Generation and Translation

GAN-Control: Explicitly Controllable GANs

We present a framework for training GANs with explicit control over generated images. We are able to control the generated image by settings exact attributes such as age, pose, expression, etc. Most approaches for editing GAN-generated images achieve partial control by leveraging the latent space disentanglement properties, obtained implicitly after standard GAN training. Such methods are able to change the relative intensity of certain attributes, but not explicitly set their values. Recently proposed methods, designed for explicit control over human faces, harness morphable 3D face models to…

Self-supervised Enhancement of Latent Discovery in GANs

Several methods for discovering interpretable directions in the latent space of pre-trained GANs have been proposed. Latent semantics discovered by unsupervised methods are relatively less disentangled than supervised methods since they do not use pre-trained attribute classifiers. We propose Scale Ranking Estimator (SRE), which is trained using self-supervision. SRE enhances the disentanglement in directions obtained by existing unsupervised disentanglement techniques. These directions are updated to preserve the ordering of variation within each direction in latent space. Qualitative and qua…

通过局部维度估计分析 GAN 的潜在空间

Analyzing the Latent Space of GAN through Local Dimension Estimation

本文提出预训练 GAN 中间层的局部维度估计算法，并以纯几何指标 Distortion 无监督评估潜在空间解耦程度。

Fine-Grained Face Swapping via Regional GAN Inversion

We present a novel paradigm for high-fidelity face swapping that faithfully preserves the desired subtle geometry and texture details. We rethink face swapping from the perspective of fine-grained face editing, \textit{i.e., ``editing for swapping'' (E4S)}, and propose a framework that is based on the explicit disentanglement of the shape and texture of facial components. Following the E4S principle, our framework enables both global and local swapping of facial features, as well as controlling the amount of partial swapping specified by the user. Furthermore, the E4S paradigm is inherently ca…

Toward a Controllable Disentanglement Network

This paper addresses two crucial problems of learning disentangled image representations, namely controlling the degree of disentanglement during image editing, and balancing the disentanglement strength and the reconstruction quality. To encourage disentanglement, we devise a distance covariance based decorrelation regularization. Further, for the reconstruction step, our model leverages a soft target representation combined with the latent image code. By exploring the real-valued space of the soft target representation, we are able to synthesize novel images with the designated properties. T…

Disentangled Face Attribute Editing via Instance-Aware Latent Space Search

Recent works have shown that a rich set of semantic directions exist in the latent space of Generative Adversarial Networks (GANs), which enables various facial attribute editing applications. However, existing methods may suffer poor attribute variation disentanglement, leading to unwanted change of other attributes when altering the desired one. The semantic directions used by existing methods are at attribute level, which are difficult to model complex attribute correlations, especially in the presence of attribute distribution bias in GAN's training set. In this paper, we propose a novel f…

PaDPaF：部分联邦GAN的部分解耦

PaDPaF: Partial Disentanglement with Partially-Federated GANs

提出PaDPaF架构，通过部分联邦GAN隐式解耦内容与风格，实现个性化生成和隐私保护，实验验证其有效性。

TunaGAN: Interpretable GAN for Smart Editing

In this paper, we introduce a tunable generative adversary network (TunaGAN) that uses an auxiliary network on top of existing generator networks (Style-GAN) to modify high-resolution face images according to user's high-level instructions, with good qualitative and quantitative performance. To optimize for feature disentanglement, we also investigate two different latent space that could be traversed for modification. The problem of mode collapse is characterized in detail for model robustness. This work could be easily extended to content-aware image editor based on other GANs and provide in…

Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs

The discovery of the disentanglement properties of the latent space in GANs motivated a lot of research to find the semantically meaningful directions on it. In this paper, we suggest that the disentanglement property is closely related to the geometry of the latent space. In this regard, we propose an unsupervised method for finding the semantic-factorizing directions on the intermediate latent space of GANs based on the local geometry. Intuitively, our proposed method, called Local Basis, finds the principal variation of the latent space in the neighborhood of the base latent variable. Exper…

MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation

Recent advances in generative adversarial networks (GANs) have led to remarkable achievements in face image synthesis. While methods that use style-based GANs can generate strikingly photorealistic face images, it is often difficult to control the characteristics of the generated faces in a meaningful and disentangled way. Prior approaches aim to achieve such semantic control and disentanglement within the latent space of a previously trained GAN. In contrast, we propose a framework that a priori models physical attributes of the face such as 3D shape, albedo, pose, and lighting explicitly, th…

Disentangled Image Generation Through Structured Noise Injection

We explore different design choices for injecting noise into generative adversarial networks (GANs) with the goal of disentangling the latent space. Instead of traditional approaches, we propose feeding multiple noise codes through separate fully-connected layers respectively. The aim is restricting the influence of each noise code to specific parts of the generated image. We show that disentanglement in the first layer of the generator network leads to disentanglement in the generated image. Through a grid-based structure, we achieve several aspects of disentanglement without complicating the…

面向高保真统一脑部MRI合成的原则性特征解耦

Principled Feature Disentanglement for High-Fidelity Unified Brain MRI Synthesis

本文提出HF-GAN统一框架，通过原则性特征解耦、多对一与一对一流及CAFF融合实现缺失多序列脑部MRI合成，并在健康与病理数据上达到最优，还可改善下游肿瘤分割。

CoDeGAN：用于生成对抗网络的对比解耦

CoDeGAN: Contrastive Disentanglement for Generative Adversarial Network

针对GAN无监督解耦中的模式崩溃问题，提出CoDeGAN，通过将解耦约束从图像域放宽至特征域并集成自监督预训练，提升了训练稳定性与解耦能力，在多个基准上优于现有方法。

Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN

Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. Yet, as most try-on approaches fit in-shop garments onto a target person, they require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability. While a few recent works attempt to transfer garments directly from one person to another, alleviating the need to collect paired datasets, their performance is impacted by the lack of paired (supervised) information. In particular, disentangling style and…

Exploring Explicit Domain Supervision for Latent Space Disentanglement in Unpaired Image-to-Image Translation

Image-to-image translation tasks have been widely investigated with Generative Adversarial Networks (GANs). However, existing approaches are mostly designed in an unsupervised manner while little attention has been paid to domain information within unpaired data. In this paper, we treat domain information as explicit supervision and design an unpaired image-to-image translation framework, Domain-supervised GAN (DosGAN), which takes the first step towards the exploration of explicit domain supervision. In contrast to representing domain characteristics using different generators or domain codes…

An Assessment of GANs for Identity-related Applications

Generative Adversarial Networks (GANs) are now capable of producing synthetic face images of exceptionally high visual quality. In parallel to the development of GANs themselves, efforts have been made to develop metrics to objectively assess the characteristics of the synthetic images, mainly focusing on visual quality and the variety of images. Little work has been done, however, to assess overfitting of GANs and their ability to generate new identities. In this paper we apply a state of the art biometric network to various datasets of synthetic images and perform a thorough assessment of th…

Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation

Unsupervised disentanglement learning is a crucial issue for understanding and exploiting deep generative models. Recently, SeFa tries to find latent disentangled directions by performing SVD on the first projection of a pre-trained GAN. However, it is only applied to the first layer and works in a post-processing way. Hessian Penalty minimizes the off-diagonal entries of the output's Hessian matrix to facilitate disentanglement, and can be applied to multi-layers.However, it constrains each entry of output independently, making it not sufficient in disentangling the latent directions (e.g., s…

Unsupervised Disentanglement of Linear-Encoded Facial Semantics

We propose a method to disentangle linear-encoded facial semantics from StyleGAN without external supervision. The method derives from linear regression and sparse representation learning concepts to make the disentangled latent representations easily interpreted as well. We start by coupling StyleGAN with a stabilized 3D deformable facial reconstruction method to decompose single-view GAN generations into multiple semantics. Latent representations are then extracted to capture interpretable facial semantics. In this work, we make it possible to get rid of labels for disentangling meaningful f…

我们永不脱离 Style：通过潜空间子空间分解实现运动解耦

We never go out of Style: Motion Disentanglement by Subspace Decomposition of Latent Space

针对视频中多成分复杂运动的解耦问题，本文利用基于 style 的 GAN 潜空间发现语义运动子空间，仅需约10个真值视频即可支持运动编辑与选择性迁移。

Multi-Attribute Balanced Sampling for Disentangled GAN Controls

Various controls over the generated data can be extracted from the latent space of a pre-trained GAN, as it implicitly encodes the semantics of the training data. The discovered controls allow to vary semantic attributes in the generated images but usually lead to entangled edits that affect multiple attributes at the same time. Supervised approaches typically sample and annotate a collection of latent codes, then train classifiers in the latent space to identify the controls. Since the data generated by GANs reflects the biases of the original dataset, so do the resulting semantic controls. W…

Information Compensation for Deep Conditional Generative Networks

In recent years, unsupervised/weakly-supervised conditional generative adversarial networks (GANs) have achieved many successes on the task of modeling and generating data. However, one of their weaknesses lies in their poor ability to separate, or disentangle, the different factors that characterize the representation encoded in their latent space. To address this issue, we propose a novel structure for unsupervised conditional GANs powered by a novel Information Compensation Connection (IC-Connection). The proposed IC-Connection enables GANs to compensate for information loss incurred during…

Y-GAN: Learning Dual Data Representations for Efficient Anomaly Detection

We propose a novel reconstruction-based model for anomaly detection, called Y-GAN. The model consists of a Y-shaped auto-encoder and represents images in two separate latent spaces. The first captures meaningful image semantics, key for representing (normal) training data, whereas the second encodes low-level residual image characteristics. To ensure the dual representations encode mutually exclusive information, a disentanglement procedure is designed around a latent (proxy) classifier. Additionally, a novel consistency loss is proposed to prevent information leakage between the latent spaces…

Optimizing Latent Space Directions For GAN-based Local Image Editing

Generative Adversarial Network (GAN) based localized image editing can suffer from ambiguity between semantic attributes. We thus present a novel objective function to evaluate the locality of an image edit. By introducing the supervision from a pre-trained segmentation network and optimizing the objective function, our framework, called Locally Effective Latent Space Direction (LELSD), is applicable to any dataset and GAN architecture. Our method is also computationally fast and exhibits a high extent of disentanglement, which allows users to interactively perform a sequence of edits on an im…

E4S：基于编辑与区域GAN反演的细粒度换脸

E4S: Fine-grained Face Swapping via Editing With Regional GAN Inversion

本文提出E4S换脸方法，通过区域GAN反演显式解耦形状与纹理，在StyleGAN潜空间中实现细粒度面部编辑，并设计重着色与修复模块优化光照与形状，显著提升纹理、形状和光照保留效果。

Interpreting the Latent Space of GANs for Semantic Face Editing

Despite the recent advance of Generative Adversarial Networks (GANs) in high-fidelity image synthesis, there lacks enough understanding of how GANs are able to map a latent code sampled from a random distribution to a photo-realistic image. Previous work assumes the latent space learned by GANs follows a distributed representation but observes the vector arithmetic phenomenon. In this work, we propose a novel framework, called InterFaceGAN, for semantic face editing by interpreting the latent semantics learned by GANs. In this framework, we conduct a detailed study on how different semantics a…

Style Intervention: How to Achieve Spatial Disentanglement with Style-based Generators?

Generative Adversarial Networks (GANs) with style-based generators (e.g. StyleGAN) successfully enable semantic control over image synthesis, and recent studies have also revealed that interpretable image translations could be obtained by modifying the latent code. However, in terms of the low-level image content, traveling in the latent space would lead to `spatially entangled changes' in corresponding images, which is undesirable in many real-world applications where local editing is required. To solve this problem, we analyze properties of the 'style space' and explore the possibility of co…

Disentangled Representation Learning Using ($β$-)VAE and GAN

Given a dataset of images containing different objects with different features such as shape, size, rotation, and x-y position; and a Variational Autoencoder (VAE); creating a disentangled encoding of these features in the hidden space vector of the VAE was the task of interest in this paper. The dSprite dataset provided the desired features for the required experiments in this research. After training the VAE combined with a Generative Adversarial Network (GAN), each dimension of the hidden vector was disrupted to explore the disentanglement in each dimension. Note that the GAN was used to im…

无跳跃连接的强化解耦人脸替换

Reinforced Disentanglement for Face Swapping without Skip Connection

针对人脸替换中身份泄露和非身份属性保留不足的问题，提出无跳跃连接框架WSC-swap，采用双目标编码器分离属性，结合GAN身份移除损失与3DMM非身份保留损失强化解耦，在多项指标上显著超越现有方法。

StyleDiffusion：通过扩散模型实现可控的解耦风格迁移

StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models

本文提出 StyleDiffusion 框架，结合 CLIP 风格解耦损失、风格重建先验与扩散模型，实现可解释、可控且效果优于现有方法的内容-风格解耦风格迁移。

CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs

Recent advances show that Generative Adversarial Networks (GANs) can synthesize images with smooth variations along semantically meaningful latent directions, such as pose, expression, layout, etc. While this indicates that GANs implicitly learn pixel-level correspondences across images, few studies explored how to extract them explicitly. In this work, we introduce Coordinate GAN (CoordGAN), a structure-texture disentangled GAN that learns a dense correspondence map for each generated image. We represent the correspondence maps of different images as warped coordinate frames transformed from …

基于多域解耦的CT扫描多视角X射线图像合成

Multi-view X-ray Image Synthesis with Multiple Domain Disentanglement from CT Scans

提出CT2X-GAN方法，通过内容和风格解耦及姿态注意力模块，从CT扫描端到端合成多视角X射线图像，在CTSpine1K数据集上表现优于3D感知方法。

PASTA-GAN++: A Versatile Framework for High-Resolution Unpaired Virtual Try-on

Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. In this work, we take a step forwards to explore versatile virtual try-on solutions, which we argue should possess three main properties, namely, they should support unsupervised training, arbitrary garment categories, and controllable garment editing. To this end, we propose a characteristic-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN++ (PASTA-GAN++), to achieve a versatile system for high-resolution unpaired virtual try-…

基于解耦隐向量的图像到图像转换用于人脸编辑

Image-to-Image Translation with Disentangled Latent Vectors for Face Editing

提出一种基于解耦隐向量的图像到图像转换框架，通过隐空间分解和正交约束学习解耦属性方向，结合注意力跳跃连接的编码器-解码器架构，显著提升了人脸属性编辑的精确性与保真度。

通过矩阵分解探索 GAN 潜在空间中的语义变化

Exploring Semantic Variations in GAN Latent Spaces via Matrix Factorization

本文通过矩阵分解探索GAN潜在空间的语义变化，发现GANSpace方法虽能生成高质量图像操作但存在高度纠缠，用ICA替换PCA可提升操作质量与分离度，且GAN规模影响图像质量但不改变基本控制方向的存在。

Learning Disentangled Representations with Latent Variation Predictability

Latent traversal is a popular approach to visualize the disentangled latent representations. Given a bunch of variations in a single unit of the latent representation, it is expected that there is a change in a single factor of variation of the data while others are fixed. However, this impressive experimental observation is rarely explicitly encoded in the objective function of learning disentangled representations. This paper defines the variation predictability of latent disentangled representations. Given image pairs generated by latent codes varying in a single dimension, this varied dime…

基于局部特征投影的三维生成模型潜在解耦

3D Generative Model Latent Disentanglement via Local Eigenprojection

本文提出一种基于谱几何的新型损失函数，通过鼓励3D网格VAEs或GANs的潜在变量遵循身份属性的局部特征投影，实现了局部形状属性的解耦生成，在保持良好生成能力和相当训练时间的同时，显著提升了潜在解耦效果。

Editing in Style: Uncovering the Local Semantics of GANs

While the quality of GAN image synthesis has improved tremendously in recent years, our ability to control and condition the output is still limited. Focusing on StyleGAN, we introduce a simple and effective method for making local, semantically-aware edits to a target output image. This is accomplished by borrowing elements from a source image, also a GAN output, via a novel manipulation of style vectors. Our method requires neither supervision from an external model, nor involves complex spatial morphing operations. Instead, it relies on the emergent disentanglement of semantic objects that …

FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery

We propose FineGAN, a novel unsupervised GAN framework, which disentangles the background, object shape, and object appearance to hierarchically generate images of fine-grained object categories. To disentangle the factors without supervision, our key idea is to use information theory to associate each factor to a latent code, and to condition the relationships between the codes in a specific way to induce the desired hierarchy. Through extensive experiments, we show that FineGAN achieves the desired disentanglement to generate realistic and diverse images belonging to fine-grained classes of …

Review of Disentanglement Approaches for Medical Applications -- Towards Solving the Gordian Knot of Generative Models in Healthcare

Deep neural networks are commonly used for medical purposes such as image generation, segmentation, or classification. Besides this, they are often criticized as black boxes as their decision process is often not human interpretable. Encouraging the latent representation of a generative model to be disentangled offers new perspectives of control and interpretability. Understanding the data generation process could help to create artificial medical data sets without violating patient privacy, synthesizing different data modalities, or discovering data generating characteristics. These character…

Feature Disentanglement in generating three-dimensional structure from two-dimensional slice with sliceGAN

Deep generative models are known to be able to model arbitrary probability distributions. Among these, a recent deep generative model, dubbed sliceGAN, proposed a new way of using the generative adversarial network (GAN) to capture the micro-structural characteristics of a two-dimensional (2D) slice and generate three-dimensional (3D) volumes with similar properties. While 3D micrographs are largely beneficial in simulating diverse material behavior, they are often much harder to obtain than their 2D counterparts. Hence, sliceGAN opens up many interesting directions of research by learning the…

Surrogate Gradient Field for Latent Space Manipulation

Generative adversarial networks (GANs) can generate high-quality images from sampled latent codes. Recent works attempt to edit an image by manipulating its underlying latent code, but rarely go beyond the basic task of attribute adjustment. We propose the first method that enables manipulation with multidimensional condition such as keypoints and captions. Specifically, we design an algorithm that searches for a new latent code that satisfies the target condition based on the Surrogate Gradient Field (SGF) induced by an auxiliary mapping network. For quantitative comparison, we propose a metr…

High-Fidelity Synthesis with Disentangled Representation

Learning disentangled representation of data without supervision is an important step towards improving the interpretability of generative models. Despite recent advances in disentangled representation learning, existing approaches often suffer from the trade-off between representation learning and generation performance i.e. improving generation quality sacrifices disentanglement performance). We propose an Information-Distillation Generative Adversarial Network (ID-GAN), a simple yet generic framework that easily incorporates the existing state-of-the-art models for both disentanglement lear…

一种用于解耦可控图像编辑的紧凑语义潜在空间

A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing

针对现有GAN图像编辑方法解耦性不足的问题，提出一种基于主成分分析压缩潜在空间的自动编码器，重构StyleGAN潜在空间以实现属性解耦编辑，训练快、解耦效果优且保持图像身份保真度，架构易实现。

Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models

Generative models have been widely studied in computer vision. Recently, diffusion models have drawn substantial attention due to the high quality of their generated images. A key desired property of image generative models is the ability to disentangle different attributes, which should enable modification towards a style without changing the semantic content, and the modification parameters should generalize to different images. Previous studies have found that generative adversarial networks (GANs) are inherently endowed with such disentanglement capability, so they can perform disentangled…

Semantic Unfolding of StyleGAN Latent Space

Generative adversarial networks (GANs) have proven to be surprisingly efficient for image editing by inverting and manipulating the latent code corresponding to an input real image. This editing property emerges from the disentangled nature of the latent space. In this paper, we identify that the facial attribute disentanglement is not optimal, thus facial editing relying on linear attribute separation is flawed. We thus propose to improve semantic disentanglement with supervision. Our method consists in learning a proxy latent representation using normalizing flows, and we show that this lead…

通过GAN潜在空间中的方向发现实现单样本神经人脸重演

One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space

提出一种利用预训练GAN潜在空间方向发现的方法，实现单样本神经人脸重演，通过3D形状模型学习控制姿态和表情的方向，在VoxCeleb基准上优于现有方法。

StyleID: Identity Disentanglement for Anonymizing Faces

Privacy of machine learning models is one of the remaining challenges that hinder the broad adoption of Artificial Intelligent (AI). This paper considers this problem in the context of image datasets containing faces. Anonymization of such datasets is becoming increasingly important due to their central role in the training of autonomous cars, for example, and the vast amount of data generated by surveillance systems. While most prior work de-identifies facial images by modifying identity features in pixel space, we instead project the image onto the latent space of a Generative Adversarial Ne…

Unsupervised Disentanglement GAN for Domain Adaptive Person Re-Identification

While recent person re-identification (ReID) methods achieve high accuracy in a supervised setting, their generalization to an unlabelled domain is still an open problem. In this paper, we introduce a novel unsupervised disentanglement generative adversarial network (UD-GAN) to address the domain adaptation issue of supervised person ReID. Our framework jointly trains a ReID network for discriminative features extraction in a source labelled domain using identity annotation, and adapts the ReID model to an unlabelled target domain by learning disentangled latent representations on the domain. …

SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing

Recent studies have shown that StyleGANs provide promising prior models for downstream tasks on image synthesis and editing. However, since the latent codes of StyleGANs are designed to control global styles, it is hard to achieve a fine-grained control over synthesized images. We present SemanticStyleGAN, where a generator is trained to model local semantic parts separately and synthesizes images in a compositional way. The structure and texture of different local parts are controlled by corresponding latent codes. Experimental results demonstrate that our model provides a strong disentanglem…

TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing

Recent advances like StyleGAN have promoted the growth of controllable facial editing. To address its core challenge of attribute decoupling in a single latent space, attempts have been made to adopt dual-space GAN for better disentanglement of style and content representations. Nonetheless, these methods are still incompetent to obtain plausible editing results with high controllability, especially for complicated attributes. In this study, we highlight the importance of interaction in a dual-space GAN for more controllable editing. We propose TransEditor, a novel Transformer-based framework …

High-resolution Face Swapping via Latent Semantics Disentanglement

We present a novel high-resolution face swapping method using the inherent prior knowledge of a pre-trained GAN model. Although previous research can leverage generative priors to produce high-resolution results, their quality can suffer from the entangled semantics of the latent space. We explicitly disentangle the latent semantics by utilizing the progressive nature of the generator, deriving structure attributes from the shallow layers and appearance attributes from the deeper ones. Identity and pose information within the structure attributes are further separated by introducing a landmark…

AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatars

Although 2D generative models have made great progress in face image generation and animation, they often suffer from undesirable artifacts such as 3D inconsistency when rendering images from different camera viewpoints. This prevents them from synthesizing video animations indistinguishable from real ones. Recently, 3D-aware GANs extend 2D GANs for explicit disentanglement of camera pose by leveraging 3D scene representations. These methods can well preserve the 3D consistency of the generated images across different views, yet they cannot achieve fine-grained control over other attributes, a…

IB-GAN：基于信息瓶颈生成对抗网络的可解耦表示学习

IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks

本文提出IB-GAN，利用信息瓶颈框架约束生成器中间层互信息，实现可解耦表示学习，在解耦得分上媲美β-VAE并优于InfoGAN，生成质量更佳。

StyleGAN 中的人脸身份感知解耦

Face Identity-Aware Disentanglement in StyleGAN

本文提出 StyleGAN 插件 PluGeN4Faces，利用电影帧中同一人物的多姿态图像和对比损失实现人脸属性与身份解耦，使属性编辑对身份等其余特征的干扰显著低于现有方法。