Cycle Encoding of a StyleGAN Encoder for Improved Reconstruction and Editability
Sun Yat-sen University（中山大学）；Xiamen University（厦门大学）；City University of Hong Kong（香港城市大学）；Guangdong University of Technology（广东工业大学）；Hong Kong Polytechnic University（香港理工大学）
Xudong Mao, Liujuan Cao, Aurele T. Gnanha, Zhenguo Yang, Qing Li, Rongrong Ji
GAN inversion aims to invert an input image into the latent space of a pre-trained GAN. Despite the recent advances in GAN inversion, there remain challenges to mitigate the tradeoff between distortion and editability, i.e. reconstructing the input image accurately and editing the inverted image with a small visual quality drop. The recently proposed pivotal tuning model makes significant progress towards reconstruction and editability, by using a two-step approach that first inverts the input image into a latent code, called pivot code, and then alters the generator so that the input image can be accurately mapped into the pivot code. Here, we show that both reconstruction and editability can be improved by a proper design of the pivot code. We present a simple yet effective method, named cycle encoding, for a high-quality pivot code. The key idea of our method is to progressively train an encoder in varying spaces according to a cycle scheme: W->W+->W. This training methodology preserves the properties of both W and W+ spaces, i.e. high editability of W and low distortion of W+. To further decrease the distortion, we also propose to refine the pivot code with an optimization-based method, where a regularization term is introduced to reduce the degradation in editability. Qualitative and quantitative comparisons to several state-of-the-art methods demonstrate the superiority of our approach.
StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tencent AI Lab（腾讯人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Editable Noise Map Inversion: Encoding Target-image into Noise For High-Fidelity Image Manipulation
Department of Artificial Intelligence, University of Hanyang, Seoul, Korea（人工智能系，翰阳大学，韩国首尔）；Department of Computer Science, University of Hanyang, Seoul, Korea（计算机科学系，翰阳大学，韩国首尔）
Affine-Equivariant Kernel Space Encoding for NeRF Editing
Poznan University of Technology, Institute of Robotics and Machine Intelligence（波兹南技术大学机器人与机器智能研究所）；Jagiellonian University, Faculty of Mathematics and Computer Science（雅盖隆大学数学与计算机科学学院）；IDEAS Research Institute（IDEAS研究所）
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
University of Texas at Austin（德克萨斯大学奥斯汀分校）；UC Santa Barbara（加州大学圣塔芭芭拉分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
最近对Transformers的爆炸性兴趣表明它们有潜力成为计算机视觉任务的强大“通用”模型，例如分类，检测和分割。虽然这些尝试主要研究判别模型，但我们在一些更困难的视觉任务上探索了Transformers，例如，生成对抗网络（GANs）。我们的目标是进行第一个试点研究，建立一个完全没有卷积的GAN，只使用纯粹的基于transformer的架构。我们的香草GAN架构，被称为transGAN，由一个内存友好的基于transformer的生成器，逐步增加特征分辨率，并相应地多尺度的搜索，以同时捕获语义上下文和低级别的纹理。在此基础上，我们引入了网格自关注的新模块，以进一步缓解内存瓶颈，从而将TransGAN扩展到高分辨率生成。我们还开发了一个独特的训练配方，包括一系列技术，可以减轻transGAN的训练不稳定性问题，如数据增强，修改归一化和相对位置编码。与当前使用卷积主干的最先进的GANs相比，我们最好的架构实现了极具竞争力的性能。具体来说，TransGAN在STL-10上设置了10.43的最新初始得分和18.28的FID，优于StyleGAN-V2。当涉及到更高分辨率（例如256 x 256）的生成任务时，例如在CelebA-HQ和LSUN-Church上，TransGAN继续生成具有高保真度和令人印象深刻的纹理细节的各种视觉示例。此外，我们深入研究基于transformer的生成模…
