arXiv 2310.150812024-03-27cs.CV

E4S：基于编辑与区域GAN反演的细粒度换脸

E4S: Fine-grained Face Swapping via Editing With Regional GAN Inversion

Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；South China University of Technology（华南理工大学）；The University of Hong Kong（香港大学）；Tencent AI Lab（腾讯AI实验室）

Maomao Li, Ge Yuan, Cairong Wang, Zhian Liu, Yong Zhang, Yongwei Nie, Jue Wang, Dong Xu

备注：: Project Page: https://e4s2024.github.io/ ;. arXiv admin note: text overlap with arXiv:2211.14068

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出E4S换脸方法，通过区域GAN反演显式解耦形状与纹理，在StyleGAN潜空间中实现细粒度面部编辑，并设计重着色与修复模块优化光照与形状，显著提升纹理、形状和光照保留效果。

摘要：

本文从细粒度面部编辑的角度提出了一种新颖的换脸方法，称为“编辑以换脸”（E4S）。传统换脸方法依赖全局特征提取，无法保留详细的源身份。相比之下，我们提出了一种区域GAN反演（RGI）方法，允许显式解耦形状与纹理。具体而言，E4S在预训练StyleGAN的潜空间中执行换脸，其中多尺度掩码引导编码器将每个面部组件的纹理投影为区域风格码，并通过掩码引导注入模块用风格码操作特征图。基于此解耦，换脸可简化为风格与掩码交换。此外，由于光照条件差异较大，将源皮肤迁移到目标图像可能导致光照不协调。我们提出重新着色网络，使换脸后保持目标光照条件同时保留源皮肤。进一步，为处理掩码交换中潜在的不匹配区域，设计了面部修复模块以优化面部形状。与最先进方法的广泛对比表明，E4S在纹理、形状和光照保留上优于现有方法。实现见https://github.com/e4s2024/E4S2024。

英文摘要：

This paper proposes a novel approach to face swapping from the perspective of fine-grained facial editing, dubbed "editing for swapping" (E4S). The traditional face swapping methods rely on global feature extraction and fail to preserve the detailed source identity. In contrast, we propose a Regional GAN Inversion (RGI) method, which allows the explicit disentanglement of shape and texture. Specifically, our E4S performs face swapping in the latent space of a pretrained StyleGAN, where a multi-scale mask-guided encoder is applied to project the texture of each facial component into regional style codes and a mask-guided injection module manipulating feature maps with the style codes. Based on this disentanglement, face swapping can be simplified as style and mask swapping. Besides, due to the large lighting condition gap, transferring the source skin into the target image may lead to disharmony lighting. We propose a re-coloring network to make the swapped face maintain the target lighting condition while preserving the source skin. Further, to deal with the potential mismatch areas during mask exchange, we design a face inpainting module to refine the face shape. The extensive comparisons with state-of-the-art methods demonstrate that our E4S outperforms existing methods in preserving texture, shape, and lighting. Our implementation is available at https://github.com/e4s2024/E4S2024.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。