Text-to-image (T2I) diffusion models, when fine-tuned on a few personal images, can generate visuals with a high degree of consistency. However, such fine-tuned models are not robust; they often fail to compose with concepts of pretrained model or other fine-tuned models. To address this, we propose a novel fine-tuning objective, dubbed Direct Consistency Optimization, which controls the deviation between fine-tuning and pretrained models to retain the pretrained knowledge during fine-tuning. Through extensive experiments on subject and style customization, we demonstrate that our method positions itself on a superior Pareto frontier between subject (or style) consistency and image-text alignment over all previous baselines; it not only outperforms regular fine-tuning objective in image-text alignment, but also shows higher fidelity to the reference images than the method that fine-tunes with additional prior dataset. More importantly, the models fine-tuned with our method can be merged without interference, allowing us to generate custom subjects in a custom style by composing separately customized subject and style models. Notably, we show that our approach achieves better prompt fidelity and subject fidelity than those post-optimized for merging regular fine-tuned models.
Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences(自动化研究所、中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院、中国科学院大学)
Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customization
School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);School of Business, Hohai University(河海大学商学院);School of Artificial Intelligence, Nanjing University of Information Science and Technology(南京信息工程大学人工智能学院)
Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);The Hong Kong Polytechnic University(香港理工大学);Fudan University(复旦大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所)
MotionCrafter: One-Shot Motion Customization of Diffusion Models
MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统全国重点实验室);School of AI, UCAS(中国科学院大学人工智能学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Kuaishou Technology(快手科技)
LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
Adobe Research, Romania(罗马尼亚Adobe研究院);Department of Computer Science, University of Bucharest, Romania(罗马尼亚布加勒斯特大学计算机科学系);International Computer High School of Bucharest, Romania(罗马尼亚布加勒斯特国际计算机高中)
MotionAdapter: Video Motion Transfer via Content-Aware Attention Customization
Hangzhou Dianzi University(杭州电子科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));The Hong Kong University of Science and Technology(香港科技大学);Ocean University of China(中国海洋大学);Singapore Management University(新加坡管理大学)
SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models
PRIS, School of Artificial Intelligence, Beijing University of Posts(PRIS人工智能学院,北京邮电大学);SketchX, CVSSP, University of Surrey(SketchX,计算机视觉与模式识别研究所,萨里大学)
Multi-party Collaborative Attention Control for Image Customization
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Department of Radiology, The First Affiliated Hospital of Army Medical University(中国人民解放军军医大学第一附属医院放射科)
VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models
Kim Jaechul Graduate School of AI(金宰澈人工智能研究生院);Bio and Brain Engineering(生物与脑工程系);Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST))
Redirecting the Flow: Image Customization through Attention Distribution Shift
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);School of Computer Science, Nanjing University(南京大学计算机科学与技术学院);School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院)
提出基于最大熵理论的Conditional Attention Distribution Shift方法,通过双分支架构CustomShift实现高效主题驱动图像生成,在DreamBooth和Custom101基准上优于现有方法。