arXiv 2406.044132024-07-24cs.CVcs.AI

基于属性特定提示学习的高效3D感知人脸图像编辑

Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning

Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Technology Innovation Institute（技术创新研究院）

Amandeep Kumar, Muhammad Awais, Sanath Narayan, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer

备注：: Accepted at ECCV, 2024. Amandeep Kumar and Muhammad Awais are joint first authors. More details are available at https://awaisrauf.github.io/3d_face_editing

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出基于属性特定提示学习的即插即用3D感知人脸编辑框架，通过文本驱动的潜在属性编辑器（LAE）和多种保持损失，实现跨姿态的高质量、视角一致且属性可控的人脸图像编辑。

摘要：

借助StyleGAN的表达能力和解耦的潜在空间，现有的2D方法采用文本提示来编辑具有不同属性的人脸图像。相比之下，生成不同目标姿态人脸的3D感知方法需要属性特定的分类器，为每个属性学习独立的模型权重，且无法扩展到新属性。在本研究中，我们提出了一种基于属性特定提示学习的高效、即插即用的3D感知人脸编辑框架，能够生成跨不同目标姿态且属性可控的人脸图像。为此，我们引入了一个文本驱动的、基于可学习风格token的潜在属性编辑器（LAE）。LAE利用预训练的视觉-语言模型，在任何预训练的3D感知GAN的潜在空间中寻找文本引导的属性特定编辑方向。它利用可学习的风格token和风格映射器来学习并将该编辑方向转换到3D潜在空间。为了使用多种属性训练LAE，我们使用了方向对比损失和风格token损失。此外，为了确保跨不同姿态和属性的视角一致性和身份保持，我们采用了多种3D感知的身份和姿态保持损失。我们的实验表明，我们提出的框架生成了具有3D感知和视角一致性的高质量图像，同时保持了属性特定的特征。我们在不同的人脸属性上展示了我们方法的有效性，包括头发颜色和风格、表情等。

英文摘要：

Drawing upon StyleGAN's expressivity and disentangled latent space, existing 2D approaches employ textual prompting to edit facial images with different attributes. In contrast, 3D-aware approaches that generate faces at different target poses require attribute-specific classifiers, learning separate model weights for each attribute, and are not scalable for novel attributes. In this work, we propose an efficient, plug-and-play, 3D-aware face editing framework based on attribute-specific prompt learning, enabling the generation of facial images with controllable attributes across various target poses. To this end, we introduce a text-driven learnable style token-based latent attribute editor (LAE). The LAE harnesses a pre-trained vision-language model to find text-guided attribute-specific editing direction in the latent space of any pre-trained 3D-aware GAN. It utilizes learnable style tokens and style mappers to learn and transform this editing direction to 3D latent space. To train LAE with multiple attributes, we use directional contrastive loss and style token loss. Furthermore, to ensure view consistency and identity preservation across different poses and attributes, we employ several 3D-aware identity and pose preservation losses. Our experiments show that our proposed framework generates high-quality images with 3D awareness and view consistency while maintaining attribute-specific features. We demonstrate the effectiveness of our method on different facial attributes, including hair color and style, expression, and others.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。