arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.07433cs.CVcs.AI
收藏

MagicFace:无需训练的通用风格人像个性化定制合成

MagicFace: Training-free Universal-Style Human Image Customized Synthesis

School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Data Science, Fudan University(复旦大学数据科学学院)

Yibin Wang, Weizhong Zhang, Cheng Jin

AI总结:

提出无需训练的MagicFace方法,通过粗到细的生成流程及RSA、RBA机制,实现多概念通用风格的人像个性化合成,解决了现有方法易过拟合且缺乏多概念定制灵活性的问题。

摘要:

当前的人像定制方法利用Stable Diffusion (SD)丰富的语义先验。然而,由于SD并非专为面向人类的生成而设计,这些方法通常需要在大规模数据集上进行大量微调,这使其容易过拟合,并阻碍了其以未见风格个性化个体的能力。此外,这些方法广泛聚焦于单概念人像合成,缺乏使用多个给定概念定制个体的灵活性,从而阻碍了其更广泛的实际应用。本文提出MagicFace,一种用于多概念通用风格人像个性化合成的无需训练新方法。我们的核心思想是模拟人类在给定特定概念时如何创建图像,即首先考虑概念的形状和姿态等因素建立语义布局,然后通过在像素级别与概念进行比较来优化细节。为实现这一过程,我们引入了从粗到细的生成流程,包含两个连续阶段:语义布局构建和概念特征注入。这通过我们的Reference-aware Self-Attention (RSA)和Region-grouped Blend Attention (RBA)机制实现。在第一阶段,RSA使潜在图像能够同时查询所有参考概念的特征,提取整体语义理解以促进初始语义布局的建立。在第二阶段,我们采用基于注意力的语义分割方法在每个步骤精确定位所有概念的潜在生成区域。随后,RBA将潜在图像的像素划分为语义组,每组从相应的参考概念查询细粒度特征。大量实验证明了我们MagicFace的优越性。

英文摘要:

Current human image customization methods leverage Stable Diffusion (SD) for its rich semantic prior. However, since SD is not specifically designed for human-oriented generation, these methods often require extensive fine-tuning on large-scale datasets, which renders them susceptible to overfitting and hinders their ability to personalize individuals with previously unseen styles. Moreover, these methods extensively focus on single-concept human image synthesis and lack the flexibility to customize individuals using multiple given concepts, thereby impeding their broader practical application. This paper proposes MagicFace, a novel training-free method for multi-concept universal-style human image personalized synthesis. Our core idea is to simulate how humans create images given specific concepts, i.e., first establish a semantic layout considering factors such as concepts' shape and posture, then optimize details by comparing with concepts at the pixel level. To implement this process, we introduce a coarse-to-fine generation pipeline, involving two sequential stages: semantic layout construction and concept feature injection. This is achieved by our Reference-aware Self-Attention (RSA) and Region-grouped Blend Attention (RBA) mechanisms. In the first stage, RSA enables the latent image to query features from all reference concepts simultaneously, extracting the overall semantic understanding to facilitate the initial semantic layout establishment. In the second stage, we employ an attention-based semantic segmentation method to pinpoint the latent generated regions of all concepts at each step. Following this, RBA divides the pixels of the latent image into semantic groups, with each group querying fine-grained features from the corresponding reference concept. Extensive experiments demonstrate the superiority of our MagicFace.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑