真实且高效的换脸：一种基于扩散模型的统一方法
Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models
AI总结：
针对现有换脸方法易产生伪影、在复杂场景下效果不佳的问题，提出基于扩散模型的统一换脸方案，通过重构任务形式、多步DDIM采样、CLIP特征解耦和掩码打乱技术，实现了高保真、低推理时间的换脸及头部交换效果。
摘要：
尽管换脸任务已取得可观进展，但生成真实的换脸图像仍难以实现，结果常存在伪影，在涉及大姿态变化、色彩差异和遮挡的场景中尤为突出。为解决这些问题，我们提出一种能更好利用扩散模型实现换脸的新方法，核心贡献如下：(a) 我们提出将换脸任务重构为自监督的训练时图像修复问题，在与目标图像融合的同时增强身份迁移效果。(b) 我们在训练过程中引入多步去噪扩散隐式模型（DDIM）采样，强化身份一致性和感知相似性。(c) 第三，我们引入CLIP特征解耦技术，从目标图像中提取姿态、表情和光照信息，提升保真度。(d) 此外，我们在修复训练中引入掩码打乱技术，借此构建可实现换脸的通用模型，还额外支持头部交换功能。我们的方法超越传统换脸范畴，可交换头发甚至配饰。与依赖多个现成模型的现有工作不同，我们的方法是相对统一的方案，因此不会受其他现成模型错误的影响。在FFHQ和CelebA数据集上开展的大量实验验证了该方法的有效性和鲁棒性，证明其能以极低的推理时间实现高保真、真实的换脸效果。我们的代码可在https://github.com/Sanoojan/REFace 获取。
英文摘要：
Despite promising progress in face swapping task, realistic swapped images remain elusive, often marred by artifacts, particularly in scenarios involving high pose variation, color differences, and occlusion. To address these issues, we propose a novel approach that better harnesses diffusion models for face-swapping by making following core contributions. (a) We propose to re-frame the face-swapping task as a self-supervised, train-time inpainting problem, enhancing the identity transfer while blending with the target image. (b) We introduce a multi-step Denoising Diffusion Implicit Model (DDIM) sampling during training, reinforcing identity and perceptual similarities. (c) Third, we introduce CLIP feature disentanglement to extract pose, expression, and lighting information from the target image, improving fidelity. (d) Further, we introduce a mask shuffling technique during inpainting training, which allows us to create a so-called universal model for swapping, with an additional feature of head swapping. Ours can swap hair and even accessories, beyond traditional face swapping. Unlike prior works reliant on multiple off-the-shelf models, ours is a relatively unified approach and so it is resilient to errors in other off-the-shelf models. Extensive experiments on FFHQ and CelebA datasets validate the efficacy and robustness of our approach, showcasing high-fidelity, realistic face-swapping with minimal inference time. Our code is available at https://github.com/Sanoojan/REFace.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
