In recent years, 3D vision has become a crucial field within computer vision, powering a wide range of applications such as autonomous driving, robotics, augmented reality, and medical imaging. This field relies on accurate perception, understanding, and reconstruction of 3D scenes from 2D images or text data sources. Diffusion models, originally designed for 2D generative tasks, offer the potential for more flexible, probabilistic methods that can better capture the variability and uncertainty present in real-world 3D data. In this paper, we review the state-of-the-art methods that use diffusion models for 3D visual tasks, including but not limited to 3D object generation, shape completion, point-cloud reconstruction, and scene construction. We provide an in-depth discussion of the underlying mathematical principles of diffusion models, outlining their forward and reverse processes, as well as the various architectural advancements that enable these models to work with 3D datasets. We also discuss the key challenges in applying diffusion models to 3D vision, such as handling occlusions and varying point densities, and the computational demands of high-dimensional data. Finally, we discuss potential solutions, including improving computational efficiency, enhancing multimodal fusion, and exploring the use of large-scale pretraining for better generalization across 3D tasks. This paper serves as a foundation for future exploration and development in this rapidly evolving field.
Diffusion Model is a Good Pose Estimator from 3D RF-Vision
School of Electrical and Electronic Engineering(电气与电子工程学院);School of Mechanical and Aerospace Engineering(机械与航空航天工程学院);Nanyang Technological University(南洋理工大学)
CLAD: Constrained Latent Action Diffusion for Vision-Language Procedure Planning
Machine Perception and Interaction group, Örebro University(机器感知与交互组,奥雷布罗大学);Collaborative Artificial Intelligence group, University of Stuttgart(协作人工智能组,斯图加特大学)
Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
King Abdullah University of Science and Technology(国王阿卜杜勒-阿齐兹大学科学与技术大学);Visual Recognition Group(视觉识别组);Faculty of Electrical Engineering(电气工程学院);Czech Technical University in Prague(布拉格捷克技术大学)
Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education(脑机智能技术教育部重点实验室)
MFVLR: Multi-domain Fine-grained Vision-Language Reconstruction for Generalizable Diffusion Face Forgery Detection and Localization
Faculty of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences)(计算机科学与技术学院,齐鲁工业大学(山东省科学院));School of Computing, National University of Singapore(国立新加坡大学计算机学院);Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences)(山东省人工智能研究院,齐鲁工业大学(山东省科学院));Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin University of Technology(教育部计算机视觉与系统重点实验室,天津工业大学)
FiT: Flexible Vision Transformer for Diffusion Model
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Sydney University(悉尼大学);The University of Hong Kong(香港大学)
Tsinghua University(清华大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Duke University(杜克大学);Shanghai Jiao Tong University(上海交通大学);Harbin Institute of Technology(哈尔滨工业大学);Tianyijiaotong Technology Ltd.(天宜交通科技有限公司)
Amortizing intractable inference in diffusion models for vision, language, and control
Mila(米拉研究所);Université de Montréal(蒙特利尔大学);KAIST(韩国科学技术院);Jagiellonian University(雅盖隆大学);Ciela Institute(谢拉研究所);Dreamfold(Dreamfold公司);Recursion(Recursion公司);CIFAR(加拿大高等研究院);University of Edinburgh(爱丁堡大学)
HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion
Research Center for Medical AI, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(医学人工智能研究中心,深圳先进技术研究所,中国科学院);Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(生物医学成像研究中心,深圳先进技术研究所,中国科学院)
The University of Hong Kong(香港大学);Tsinghua University(清华大学);Duke University(杜克大学);University of Rochester(罗切斯特大学);The Ohio State University(俄亥俄州立大学);Bytedance(字节跳动);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Apple(苹果公司);The Hong Kong Polytechnic University(香港理工大学);Princeton University(普林斯顿大学)
Diffusion Models: A Comprehensive Survey of Methods and Applications
Peking University(北京大学);OpenAI;University of California, Los Angeles(加州大学洛杉矶分校);Carnegie Mellon University(卡内基梅隆大学);University of California at Merced(加州大学默塞德分校)