Recent advances in diffusion models attempt to handle conditional generative tasks by utilizing a differentiable loss function for guidance without the need for additional training. While these methods achieved certain success, they often compromise on sample quality and require small guidance step sizes, leading to longer sampling processes. This paper reveals that the fundamental issue lies in the manifold deviation during the sampling process when loss guidance is employed. We theoretically show the existence of manifold deviation by establishing a certain lower bound for the estimation error of the loss guidance. To mitigate this problem, we propose Diffusion with Spherical Gaussian constraint (DSG), drawing inspiration from the concentration phenomenon in high-dimensional Gaussian distributions. DSG effectively constrains the guidance step within the intermediate data manifold through optimization and enables the use of larger guidance steps. Furthermore, we present a closed-form solution for DSG denoising with the Spherical Gaussian constraint. Notably, DSG can seamlessly integrate as a plugin module within existing training-free conditional diffusion methods. Implementing DSG merely involves a few lines of additional code with almost no extra computational overhead, yet it leads to significant performance improvements. Comprehensive experimental results in various conditional generation tasks validate the superiority and adaptability of DSG in terms of both sample quality and time efficiency.
How much explicit guidance is necessary for conditional diffusion? We consider the problem of conditional sampling using an unconditional diffusion model and limited explicit guidance (e.g., a noised classifier, or a conditional diffusion model) that is restricted to a small number of time steps. We explore a model predictive control (MPC)-like approach to approximate guidance by simulating unconditional diffusion forward, and backpropagating explicit guidance feedback. MPC-approximated guides have high cosine similarity to real guides, even over large simulation distances. Adding MPC steps im…
Opening the Black Box of Classifier-Free Guidance via Information Bottleneck
Shanghai Jiao Tong University（上海交通大学）；vivo Mobile Communication Co., Ltd.（维沃移动通信有限公司）；Stanford University（斯坦福大学）；University of British Columbia（不列颠哥伦比亚大学）
Classifier guidance is a recently introduced method to trade off mode coverage and sample fidelity in conditional diffusion models post training, in the same spirit as low temperature sampling or truncation in other types of generative models. Classifier guidance combines the score estimate of a diffusion model with the gradient of an image classifier and thereby requires training an image classifier separate from the diffusion model. It also raises the question of whether guidance can be performed without a classifier. We show that guidance can be indeed performed by a pure generative model w…
Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学（KAUST））；Center for Research and Formation in Artificial Intelligence, Universidad de los Andes（安第斯大学人工智能研究与培训中心）；GenAI, Meta（Meta 生成式人工智能部门）
Duke University（杜克大学）；Duke University School of Medicine（杜克大学医学院）
Recent advancements in conditional Generative Adversarial Networks (cGANs) have shown promises in label guided image synthesis. Semantic masks, such as sketches and label maps, are another intuitive and effective form of guidance in image synthesis. Directly incorporating the semantic masks as constraints dramatically reduces the variability and quality of the synthesized results. We observe this is caused by the incompatibility of features from different inputs (such as mask image and latent vector) of the generator. To use semantic masks as guidance whilst providing realistic synthesized res…
Diffusion Models with Double Guidance: Generate with aggregated datasets
Graduate University of Advanced Studies (SOKENDAI), Kanagawa, Japan（综合研究大学院大学（SOKENDAI），神奈川，日本）；The Institute of Statistical Mathematics, Tokyo, Japan（统计数理研究所，东京，日本）
Spherical Brownian Bridge Diffusion Models for Conditional Cortical Thickness Forecasting
Lab for AI in Medical Imaging, Technical University of Munich, Munich, Germany（人工智能医学影像实验室，慕尼黑技术大学，慕尼黑，德国）；Munich Center for Machine Learning, Munich, Germany（慕尼黑机器学习中心，慕尼黑，德国）
Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifier-free guidance are favored…
Diffusion Models for Solving Inverse Problems via Posterior Sampling with Piecewise Guidance
Department of Electrical and Computer Engineering（电气与计算机工程系）；Virginia Tech（弗吉尼亚理工大学）；Department of Electrical and Electronic Engineering（电气电子工程系）；Institute of Science（科学研究所）
GradCheck: Analyzing classifier guidance gradients for conditional diffusion sampling
Center for Artificial Intelligence, Technical University of Applied Sciences Wuerzburg-Schweinfurt（维尔茨堡-施韦因富特应用技术大学人工智能中心）；Bielefeld University（比勒费尔德大学）
SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Score
Department of Computer Science and Engineering, The Chinese University of Hong Kong（计算机科学与工程系，香港中文大学）；Department of Information Engineering, The Chinese University of Hong Kong（信息工程系，香港中文大学）
Spherical images taken in all directions (360 degrees) allow representing the surroundings of the subject and the space itself, providing an immersive experience to the viewers. Generating a spherical image from a single normal-field-of-view (NFOV) image is convenient and considerably expands the usage scenarios because there is no need to use a specific panoramic camera or take images from multiple directions; however, it is still a challenging and unsolved problem. The primary challenge is controlling the high degree of freedom involved in generating a wide area that includes the all directi…
Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation
Nanyang Technological University（南洋理工大学）；South China University of Technology（华南理工大学）；University of Oxford（牛津大学）；The University of Sydney（悉尼大学）；JD Explore Academy（京东探索研究院）
CoPHo: Classifier-guided Conditional Topology Generation with Persistent Homology
School of Cyberspace Security, Beijing University of Posts and Telecommunications（网络安全学院，北京邮电大学）；State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（网络与交换技术国家重点实验室，北京邮电大学）；School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications（计算机学院（国家级试点软件工程学院），北京邮电大学）
