基于扩散模型的化妆转移与面部区域感知化妆特征
Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features
AI总结：
本文提出FRAM方法，通过微调化妆CLIP和区域感知化妆注入，提升扩散模型在面部区域特定化妆转移中的可控性与效果。
摘要：
当前基于扩散模型的化妆转移方法通常使用现成的基础模型（如CLIP）编码的化妆信息作为条件，以保持参考图像的化妆风格。尽管有效，这些方法主要存在两个局限：（1）为通用任务预训练的基础模型难以捕捉化妆风格；（2）参考图像的化妆特征被注入扩散去噪模型作为整体进行全局化妆转移，忽略了面部区域感知的化妆特征（如眼睛、嘴巴等），限制了区域特定化妆转移的可控性。为此，本文提出面部区域感知化妆特征（FRAM），包含两个阶段：（1）化妆CLIP微调；（2）身份和面部区域感知化妆注入。在化妆CLIP微调中，不同于先前工作使用现成的CLIP，我们利用GPT-o3和文本驱动图像编辑模型合成标注的化妆风格数据，并使用这些数据通过自监督和图像-文本对比学习训练化妆CLIP编码器。在身份和面部区域感知化妆注入中，我们从阶段1编辑的前后化妆图像对中构建，并利用它们学习将源图像的身份和参考图像的化妆注入到扩散去噪模型中进行化妆转移。具体而言，我们使用可学习的标记查询化妆CLIP编码器以提取面部区域感知的化妆特征进行化妆注入，通过注意力损失学习以实现区域控制。至于身份注入，我们使用ControlNet联合编码源图像及其3D网格。实验结果验证了我们区域可控性和化妆转移性能的优越性。代码可在https://github.com/zaczgao/Facial_Region-Aware_Makeup获取。
英文摘要：
Current diffusion-based makeup transfer methods commonly use the makeup information encoded by off-the-shelf foundation models (e.g., CLIP) as condition to preserve the makeup style of reference image in the generation. Although effective, these works mainly have two limitations: (1) foundation models pre-trained for generic tasks struggle to capture makeup styles; (2) the makeup features of reference image are injected to the diffusion denoising model as a whole for global makeup transfer, overlooking the facial region-aware makeup features (i.e., eyes, mouth, etc) and limiting the regional controllability for region-specific makeup transfer. To address these, in this work, we propose Facial Region-Aware Makeup features (FRAM), which has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection. For makeup CLIP fine-tuning, unlike prior works using off-the-shelf CLIP, we synthesize annotated makeup style data using GPT-o3 and text-driven image editing model, and then use the data to train a makeup CLIP encoder through self-supervised and image-text contrastive learning. For identity and facial region-aware makeup injection, we construct before-and-after makeup image pairs from the edited images in stage 1 and then use them to learn to inject identity of source image and makeup of reference image to the diffusion denoising model for makeup transfer. Specifically, we use learnable tokens to query the makeup CLIP encoder to extract facial region-aware makeup features for makeup injection, which is learned via an attention loss to enable regional control. As for identity injection, we use a ControlNet Union to encode source image and its 3D mesh simultaneously. The experimental results verify the superiority of our regional controllability and our makeup transfer performance. Code is available at https://github.com/zaczgao/Facial_Region-Aware_Makeup.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
