Denoising diffusion models represent a recent emerging topic in computer vision, demonstrating remarkable results in the area of generative modeling. A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. In the forward diffusion stage, the input data is gradually perturbed over several steps by adding Gaussian noise. In the reverse stage, a model is tasked at recovering the original input data by learning to gradually reverse the diffusion process, step by step. Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens, i.e. low speeds due to the high number of steps involved during sampling. In this survey, we provide a comprehensive review of articles on denoising diffusion models applied in vision, comprising both theoretical and practical contributions in the field. First, we identify and present three generic diffusion modeling frameworks, which are based on denoising diffusion probabilistic models, noise conditioned score networks, and stochastic differential equations. We further discuss the relations between diffusion models and other deep generative models, including variational auto-encoders, generative adversarial networks, energy-based models, autoregressive models and normalizing flows. Then, we introduce a multi-perspective categorization of diffusion models applied in computer vision. Finally, we illustrate the current limitations of diffusion models and envision some interesting directions for future research.
Tsinghua University(清华大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Duke University(杜克大学);Shanghai Jiao Tong University(上海交通大学);Harbin Institute of Technology(哈尔滨工业大学);Tianyijiaotong Technology Ltd.(天宜交通科技有限公司)
Diffusion Models: A Comprehensive Survey of Methods and Applications
Peking University(北京大学);OpenAI;University of California, Los Angeles(加州大学洛杉矶分校);Carnegie Mellon University(卡内基梅隆大学);University of California at Merced(加州大学默塞德分校)
Amortizing intractable inference in diffusion models for vision, language, and control
Mila(米拉研究所);Université de Montréal(蒙特利尔大学);KAIST(韩国科学技术院);Jagiellonian University(雅盖隆大学);Ciela Institute(谢拉研究所);Dreamfold(Dreamfold公司);Recursion(Recursion公司);CIFAR(加拿大高等研究院);University of Edinburgh(爱丁堡大学)
HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion
Research Center for Medical AI, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(医学人工智能研究中心,深圳先进技术研究所,中国科学院);Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(生物医学成像研究中心,深圳先进技术研究所,中国科学院)
Diffusion Model is a Good Pose Estimator from 3D RF-Vision
School of Electrical and Electronic Engineering(电气与电子工程学院);School of Mechanical and Aerospace Engineering(机械与航空航天工程学院);Nanyang Technological University(南洋理工大学)
CLAD: Constrained Latent Action Diffusion for Vision-Language Procedure Planning
Machine Perception and Interaction group, Örebro University(机器感知与交互组,奥雷布罗大学);Collaborative Artificial Intelligence group, University of Stuttgart(协作人工智能组,斯图加特大学)
Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
King Abdullah University of Science and Technology(国王阿卜杜勒-阿齐兹大学科学与技术大学);Visual Recognition Group(视觉识别组);Faculty of Electrical Engineering(电气工程学院);Czech Technical University in Prague(布拉格捷克技术大学)
Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education(脑机智能技术教育部重点实验室)
MFVLR: Multi-domain Fine-grained Vision-Language Reconstruction for Generalizable Diffusion Face Forgery Detection and Localization
Faculty of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences)(计算机科学与技术学院,齐鲁工业大学(山东省科学院));School of Computing, National University of Singapore(国立新加坡大学计算机学院);Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences)(山东省人工智能研究院,齐鲁工业大学(山东省科学院));Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin University of Technology(教育部计算机视觉与系统重点实验室,天津工业大学)
FiT: Flexible Vision Transformer for Diffusion Model
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Sydney University(悉尼大学);The University of Hong Kong(香港大学)
The Ohio State University(俄亥俄州立大学);Indiana University(印第安纳大学);Fudan University(复旦大学);Hangzhou City University(杭州城市大学);The University of Hong Kong(香港大学);Tongji University(同济大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学)
The University of Hong Kong(香港大学);Tsinghua University(清华大学);Duke University(杜克大学);University of Rochester(罗切斯特大学);The Ohio State University(俄亥俄州立大学);Bytedance(字节跳动);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Apple(苹果公司);The Hong Kong Polytechnic University(香港理工大学);Princeton University(普林斯顿大学)