Text-to-image diffusion models have shown remarkable success in generating personalized subjects based on a few reference images. However, current methods often fail when generating multiple subjects simultaneously, resulting in mixed identities with combined attributes from different subjects. In this work, we present MuDI, a novel framework that enables multi-subject personalization by effectively decoupling identities from multiple subjects. Our main idea is to utilize segmented subjects generated by a foundation model for segmentation (Segment Anything) for both training and inference, as a form of data augmentation for training and initialization for the generation process. Moreover, we further introduce a new metric to better evaluate the performance of our method on multi-subject personalization. Experimental results show that our MuDI can produce high-quality personalized images without identity mixing, even for highly similar subjects as shown in Figure 1. Specifically, in human evaluation, MuDI obtains twice the success rate for personalizing multiple subjects without identity mixing over existing baselines and is preferred over 70% against the strongest baseline.
Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalization
Sun Yat-sen University(中山大学);East China Normal University(华东师范大学);Guangdong University of Technology(广东工业大学);The Hong Kong Polytechnic University(香港理工大学)
Meta-LoRA: Meta-Learning LoRA Components for Domain-Aware ID Personalization
Pixery Labs İstanbul, Türkiye(Pixery实验室伊斯坦布尔,土耳其);METU Dept. of Computer Engineering(METU计算机工程系);METU Dept. of Electrical and Elec. Engineering(METU电气与电子工程系)
When Identities Collapse: A Stress-Test Benchmark for Multi-Subject Personalization
University of California, Los Angeles(加州大学洛杉矶分校);University of Southern California(南加州大学);DeerLab LLC(DeerLab有限责任公司);Carnegie Mellon University(卡内基梅隆大学)
PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors
Institute of Information Engineering, Chinese Academy of Sciences(信息工程研究所,中国科学院);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);BraneMatrix AI;School of Cyber Science and Technology, Shenzhen Campus, Sun Yat-sen University(中山大学深圳校区计算机科学与技术学院)
Customizing Video Portraits via Identity-ActionDecoupling
College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);The University of Hong Kong(香港大学)
DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Intel Labs China(英特尔中国研究院);Tsinghua University(清华大学)
DreamMix: Decoupling Object Attributes for Enhanced Editability in Customized Image Inpainting
Information and Communication Engineering(信息与通信工程);Dalian University of Technology(大连理工大学);ZMO.AI Inc.(ZMO.AI公司);University of Electronic Science and Technology of China(电子科技大学)
TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究所,中国科学院);Northeastern University(东北大学);Shenzhen University of Advanced Technology(深圳先进技术大学)
OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Tencent AI Lab(腾讯AI实验室);International Digital Economy Academy(国际数字经济学院);Nanjing University(南京大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Shenzhen University(深圳大学);The Hong Kong University of Science and Technology(香港科技大学)