CLIP-PAE: 用于提取相关特征以实现解耦、可解释且可控的文本引导人脸编辑的投影增强嵌入
CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation
AI总结：
针对CLIP联合空间中文本与图像嵌入差异导致的文本引导图像编辑伪影多、解耦性等不足，提出投影增强嵌入PAE作为优化目标，可集成到各类CLIP编辑算法中，在人脸编辑任务上实现了更优的解耦、可解释与可控效果及顶尖性能。
摘要：
近期提出的对比语言-图像预训练（Contrastive Language-Image Pre-Training, CLIP）通过将图像和文本嵌入到联合潜在空间中，在二者之间搭建起桥梁，为大量旨在通过文本描述编辑输入图像的研究开辟了道路。然而，由于联合空间中图像嵌入与文本嵌入存在差异，将文本嵌入作为优化目标通常会在生成的图像中引入不必要的伪影，同时编辑过程的解耦性、可解释性与可控性也难以保证。为缓解这些问题，我们提出定义由相关提示（prompt）张成的语料子空间，以捕捉特定的图像特征；并提出CLIP投影增强嵌入（Projection-Augmentation Embedding, PAE）作为优化目标，以提升文本引导图像编辑的性能。我们的方法是一种简单通用的范式，计算与适配都十分简便，可平滑集成到任何基于CLIP的图像编辑算法中。为验证方法的有效性，我们开展了多项理论与实证研究；作为案例分析，我们将该方法应用于文本引导的语义人脸编辑任务，通过定量与定性实验证明，PAE能实现更具解耦性、可解释性与可控性的图像编辑，且质量与精度达到当前最优水平。项目主页：https://chenliang-zhou.github.io/CLIP-PAE/。
英文摘要：
Recently introduced Contrastive Language-Image Pre-Training (CLIP) bridges images and text by embedding them into a joint latent space. This opens the door to ample literature that aims to manipulate an input image by providing a textual explanation. However, due to the discrepancy between image and text embeddings in the joint space, using text embeddings as the optimization target often introduces undesired artifacts in the resulting images. Disentanglement, interpretability, and controllability are also hard to guarantee for manipulation. To alleviate these problems, we propose to define corpus subspaces spanned by relevant prompts to capture specific image characteristics. We introduce CLIP Projection-Augmentation Embedding (PAE) as an optimization target to improve the performance of text-guided image manipulation. Our method is a simple and general paradigm that can be easily computed and adapted, and smoothly incorporated into any CLIP-based image manipulation algorithm. To demonstrate the effectiveness of our method, we conduct several theoretical and empirical studies. As a case study, we utilize the method for text-guided semantic face editing. We quantitatively and qualitatively demonstrate that PAE facilitates a more disentangled, interpretable, and controllable image manipulation with state-of-the-art quality and accuracy. Project page: https://chenliang-zhou.github.io/CLIP-PAE/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
