TLCM: Training-efficient Latent Consistency Model for Image Generation with 2-8 Steps
AI Center, Guangdong OPPO Mobile Telecommunications Corp., Ltd(广东欧珀移动通信有限公司AI中心);Qing Yuan Research Institute, SEIEE, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院清源研究院)
Qingsong Xie, Zhenyi Liao, Zhijie Deng, Chen chen, Haonan Lu
Distilling latent diffusion models (LDMs) into ones that are fast to sample from is attracting growing research interest. However, the majority of existing methods face two critical challenges: (1) They hinge on long training using a huge volume of real data. (2) They routinely lead to quality degradation for generation, especially in text-image alignment. This paper proposes a novel training-efficient Latent Consistency Model (TLCM) to overcome these challenges. Our method first accelerates LDMs via data-free multistep latent consistency distillation (MLCD), and then data-free latent consistency distillation is proposed to efficiently guarantee the inter-segment consistency in MLCD. Furthermore, we introduce bags of techniques, e.g., distribution matching, adversarial learning, and preference learning, to enhance TLCM's performance at few-step inference without any real data. TLCM demonstrates a high level of flexibility by enabling adjustment of sampling steps within the range of 2 to 8 while still producing competitive outputs compared to full-step approaches. Notably, TLCM enjoys the data-free merit by employing synthetic data from the teacher for distillation. With just 70 training hours on an A100 GPU, a 3-step TLCM distilled from SDXL achieves an impressive CLIP Score of 33.68 and an Aesthetic Score of 5.97 on the MSCOCO-2017 5K benchmark, surpassing various accelerated models and even outperforming the teacher model in human preference metrics. We also demonstrate the versatility of TLCMs in applications including image style transfer, controllable generation, and Chinese-to-image generation.
Multi-scale Latent Point Consistency Models for 3D Shape Generation
Wangxuan Institute of Computer Technology, Peking University(北京大学王轩计算机技术研究所);Department of Electrical and Computer Engineering, University of British Columbia(不列颠哥伦比亚大学电气与计算机工程系)
Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping
South China University of Technology(华南理工大学);Nanyang Technological University(南洋理工大学);Beijing Institute of Technology(北京理工大学);The University of Sydney(悉尼大学)
LDC-VAE: A Latent Distribution Consistency Approach to Variational AutoEncoders
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
InterLCM: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration
Nankai University(南开大学);Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Linkoping University(林雪平大学);SB Intuitions, SoftBank(软银SB Intuitions);Nankai International Advanced Research Institute (Shenzhen Futian), Nankai University(南开大学深圳福田国际先进研究院)
LATINO-PRO: LAtent consisTency INverse sOlver with PRompt Optimization
CNRS(法国国家科学研究中心);Univ. Bordeaux(波尔多大学);INRIA(法国国家信息与自动化研究所);Bordeaux INP(波尔多国立高等工程技术学院);Heriot-Watt University(赫瑞瓦特大学);Maxwell Institute for Mathematical Sciences(麦克斯韦数学科学研究所)
DreamLCM: Towards High-Quality Text-to-3D Generation via Latent Consistency Model
Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学研究所);Visual Intelligence + X International Joint Laboratory of the Ministry of Education(教育部视觉智能与X国际联合实验室);College of Electrical Engineering and Automation, Shandong University of Science and Technology(山东科技大学电气与自动化工程学院);PengCheng Laboratory(鹏城实验室)
Visual Prompting for One-shot Controllable Video Editing without Inversion
Singapore University of Technology and Design(新加坡科技设计大学);Wuhan University(武汉大学);Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore(新加坡资讯与通信研究院)
SurgicaL-CD: Generating Surgical Images via Unpaired Image Translation with Latent Consistency Diffusion Models
NCT/UCC Dresden(德累斯顿癌症中心/癌症联盟);DKFZ Heidelberg(海德堡德国癌症研究中心);Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology(德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯医学院及大学医院);HZDR Dresden(德累斯顿亥姆霍兹中心);The Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology(德累斯顿工业大学人机交互触觉互联网中心)
Zero-Shot Video Restoration and Enhancement with Text-to-Image Latent Diffusion Models and Multi-Modal References
School of Electrical and Information Engineering, Tianjin University(天津大学电气自动化与信息工程学院);Lappeenranta-Lahti University of Technology LUT(拉彭兰塔-拉赫蒂理工大学(LUT))
PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models
Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Dalian University of Technology(大连理工大学);The University of Hong Kong(香港大学);IIIS, Tsinghua University(清华大学智能产业研究院);Hugging Face
LatentColorization: Latent Diffusion-Based Speaker Video Colorization
SFI Centre for Research Training in Artificial Intelligence(SFI人工智能研究培训中心);University of Galway(戈尔韦大学);School of Computer Science(计算机科学学院);School of Engineering(工程学院)
VideoWeave: Unlocking Geometric Consistency in Video Generation via Joint Geometry-Video Modeling
Nanjing University(南京大学);Institute of Artificial Intelligence, China Telecom (TeleAI)(中国电信人工智能研究院(TeleAI));Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Consistency Regularised Gradient Flows for Inverse Problems
Laboratoire MAP5, UMR 8145, Université Paris Cité, CNRS(巴黎城市大学MAP5实验室,UMR 8145,CNRS);Department of Mathematics, Imperial College London(伦敦帝国理工学院数学系);Heriot-Watt University, MACS & Maxwell Institute(赫瑞瓦德大学,MACS与麦克斯韦研究所)
LVMark: Robust Watermark for Latent Video Diffusion Models
Department of Artificial Intelligence, Korea University, Seoul 02841, Republic of Korea(人工智能系,韩国大学,首尔02841,大韩民国);Google DeepMind, Mountain View, CA 94043, USA(谷歌DeepMind,山景城,加利福尼亚州94043,美国)
Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification
University of Amsterdam(阿姆斯特丹大学);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Inception Institute of Artificial Intelligence(起源人工智能研究院)