arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.04243cs.CVcs.AI
收藏

面向文本到图像模型多主体个性化的身份解耦

Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models

KAIST(韩国科学技术院)

Sangwon Jang, Jaehyeong Jo, Kimin Lee, Sung Ju Hwang

AI总结:

本文提出 MuDI 框架,利用 Segment Anything 的分割主体进行训练增强与生成初始化,以解耦多主体身份,并引入新评估指标,显著降低身份混合。

摘要:

文本到图像扩散模型在基于少量参考图像生成个性化主体方面已展现出显著成功。然而,当前方法在同时生成多个主体时往往会失败,导致身份混合,即不同主体的属性被组合在一起。在本工作中,我们提出 MuDI,这是一个通过有效解耦来自多个主体的身份来实现多主体个性化的新颖框架。我们的主要思想是将分割基础模型 Segment Anything 生成的分割主体同时用于训练和推理:在训练中作为一种数据增强形式,并在生成过程中用作初始化。此外,我们进一步引入了一项新指标,以更好地评估我们的方法在多主体个性化上的性能。实验结果表明,我们的 MuDI 能够生成没有身份混合的高质量个性化图像,即使对于高度相似的主体也是如此,如图 1 所示。具体而言,在人工评估中,MuDI 在不发生身份混合的情况下个性化多个主体的成功率是现有基线的两倍,并且相对于最强基线,其偏好率超过 70%。

英文摘要:

Text-to-image diffusion models have shown remarkable success in generating personalized subjects based on a few reference images. However, current methods often fail when generating multiple subjects simultaneously, resulting in mixed identities with combined attributes from different subjects. In this work, we present MuDI, a novel framework that enables multi-subject personalization by effectively decoupling identities from multiple subjects. Our main idea is to utilize segmented subjects generated by a foundation model for segmentation (Segment Anything) for both training and inference, as a form of data augmentation for training and initialization for the generation process. Moreover, we further introduce a new metric to better evaluate the performance of our method on multi-subject personalization. Experimental results show that our MuDI can produce high-quality personalized images without identity mixing, even for highly similar subjects as shown in Figure 1. Specifically, in human evaluation, MuDI obtains twice the success rate for personalizing multiple subjects without identity mixing over existing baselines and is preferred over 70% against the strongest baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑