The use of guidance in diffusion models was originally motivated by the premise that the guidance-modified score is that of the data distribution tilted by a conditional likelihood raised to some power. In this work we clarify this misconception by rigorously proving that guidance fails to sample from the intended tilted distribution.
Our main result is to give a fine-grained characterization of the dynamics of guidance in two cases, (1) mixtures of compactly supported distributions and (2) mixtures of Gaussians, which reflect salient properties of guidance that manifest on real-world data. In both cases, we prove that as the guidance parameter increases, the guided model samples more heavily from the boundary of the support of the conditional distribution. We also prove that for any nonzero level of score estimation error, sufficiently large guidance will result in sampling away from the support, theoretically justifying the empirical finding that large guidance results in distorted generations.
In addition to verifying these results empirically in synthetic settings, we also show how our theoretical insights can offer useful prescriptions for practical deployment.
FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance
Unmanned System Research Institute, Northwestern Polytechnical University, Xi’an, China(无人系统研究院、西北工业大学、西安,中国);School of Software, Northwestern Polytechnical University, Xi’an, China(软件学院、西北工业大学、西安,中国);School of Automation, Northwestern Polytechnical University, Xi’an, China(自动化学院、西北工业大学、西安,中国);School of Astronautics, Northwestern Polytechnical University, Xi’an, China(航天学院、西北工业大学、西安,中国)
BeautyGRPO: Aesthetic Alignment for Face Retouching via Dynamic Path Guidance and Fine-Grained Preference Modeling
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, China(中山大学计算机科学与技术学院,深圳校区);vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China(vivo蓝影实验室,vivo移动通信有限公司)
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);College of Computer Science, Nankai University(南开大学计算机学院)
FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classes
School of Informatics, Xiamen University(厦门大学信息学院);Shenyang Institute of Computing Technology of Chinese Academy of Sciences(中国科学院沈阳计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);School of Mathematics and Information Engineering, Longyan University(龙岩学院数学与信息工程学院)
Chalmers University of Technology and University of Gothenburg, Dept. Computer Science and Engineering(查尔姆斯理工大学和哥德堡大学,计算机科学与工程系);Massachusetts Institute of Technology, Research Laboratory of Electronics(麻省理工学院,电子研究实验室);Technical University of Munich(慕尼黑技术大学);Helmholtz AI, Munich(海德堡人工智能研究所,慕尼黑);Max Planck Institute for Intelligent Systems, Tübingen(马克斯·普朗克智能系统研究所,图宾根)
HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidance
Sun Yat-sen University(中山大学);Xi’an Jiaotong University(西安交通大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);United Arab Emirates University(阿联酋大学)
AlbedoEdit: Unified Instance-Level Video Editing with Albedo Guidance
Max Planck Institute for Informatics(马克斯·普朗克信息研究所);University of California Santa Barbara(加州大学圣巴巴拉分校);NVIDIA Research(NVIDIA研究);Texas A & M University(德克萨斯A&M大学)
Follow-Your-MultiPose: Tuning-Free Multi-Character Text-to-Video Generation via Pose Guidance
School of Big Data and Software Engineering, Chongqing University(重庆大学大数据与软件学院);The Hong Kong University of Science and Technology(香港科技大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shanghai Jiao Tong University(上海交通大学)
Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting
Computer Aided Medical Procedures, Technische Universität München, Germany(计算机辅助医疗程序,慕尼黑技术大学,德国);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心,德国)
Make Optimization Once and for All with Fine-grained Guidance
National University of Singapore(新加坡国立大学);The Hong Kong University of Science and Technology(香港科技大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)