arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00639cs.CV
收藏

Diff-PC: 保持身份和3D感知的可控扩散用于零样本肖像定制

Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customization

School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);School of Business, Hohai University(河海大学商学院);School of Artificial Intelligence, Nanjing University of Information Science and Technology(南京信息工程大学人工智能学院)

Yifang Xu, Benxiang Zhai, Chenyu Zhang, Ming Li, Yang Li, Sidan Du

AI总结:

Diff-PC通过3D感知和可控扩散方法实现零样本肖像定制,提升身份保持和面部可控性。

摘要:

肖像定制(PC)由于其潜在应用最近引起了广泛关注。然而,现有PC方法缺乏精确的身份(ID)保持和面部控制。为了解决这些问题,我们提出了Diff-PC,一种基于扩散的框架用于零样本PC,能够生成具有高ID保真度、指定面部属性和多样化背景的逼真肖像。具体而言,我们的方法采用3D面部预测器来重建包含参考ID、目标表情和姿态的3D感知面部先验。为了捕捉细粒度面部细节,我们设计了ID编码器,用于融合局部和全局面部特征。随后,我们设计了ID控制,利用3D面部来引导ID特征的对齐。我们进一步引入ID注入器以增强ID保真度和面部可控性。最后,在我们收集的ID中心数据集上训练可以提高面部相似性和文本到图像(T2I)对齐性。广泛的实验表明,Diff-PC在ID保持、面部控制和T2I一致性方面均优于最先进的方法。此外,我们的方法与多风格基础模型兼容。

英文摘要:

Portrait customization (PC) has recently garnered significant attention due to its potential applications. However, existing PC methods lack precise identity (ID) preservation and face control. To address these tissues, we propose Diff-PC, a diffusion-based framework for zero-shot PC, which generates realistic portraits with high ID fidelity, specified facial attributes, and diverse backgrounds. Specifically, our approach employs the 3D face predictor to reconstruct the 3D-aware facial priors encompassing the reference ID, target expressions, and poses. To capture fine-grained face details, we design ID-Encoder that fuses local and global facial features. Subsequently, we devise ID-Ctrl using the 3D face to guide the alignment of ID features. We further introduce ID-Injector to enhance ID fidelity and facial controllability. Finally, training on our collected ID-centric dataset improves face similarity and text-to-image (T2I) alignment. Extensive experiments demonstrate that Diff-PC surpasses state-of-the-art methods in ID preservation, facial control, and T2I consistency. Furthermore, our method is compatible with multi-style foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑