Alignment of Diffusion Models: Fundamentals, Challenges, and Future
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Baidu Inc.(百度公司);The University of Bologna(博洛尼亚大学)
Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Zhiqiang Xu, Haoyi Xiong, James Kwok, Sumi Helal, Zeke Xie
备注:
Accepted at ACM Computing Surveys. 35 pages, 5 figures, 4 tables. Paper List: github.com/xie-lab-ml/awesome-alignment-of-diffusion-models
Diffusion models have emerged as the leading paradigm in generative modeling, excelling in various applications. Despite their success, these models often misalign with human intentions and generate results with undesired properties or even harmful content. Inspired by the success and popularity of alignment in tuning large language models, recent studies have investigated aligning diffusion models with human expectations and preferences. This work mainly reviews alignment of diffusion models, covering advancements in fundamentals of alignment, alignment techniques of diffusion models, preference benchmarks, and evaluation for diffusion models. Moreover, we discuss key perspectives on current challenges and promising future directions on solving the remaining challenges in alignment of diffusion models. To the best of our knowledge, our work is the first comprehensive review paper for researchers and engineers to comprehend, practice, and research alignment of diffusion models.
Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
University of Liverpool(利物浦大学);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Edinburgh(爱丁堡大学);University of Bristol(布里斯托大学);University of Exeter(埃克塞特大学)
Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets
Mila, Université de Montréal(蒙特利尔大学Mila实验室);Max Planck Institute for Intelligent Systems - Tübingen(智能系统马克斯·普朗克研究所(图宾根));The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));University of Tübingen(图宾根大学);University of Cambridge(剑桥大学);Microsoft Research(微软研究院)
SARA: Structural and Adversarial Representation Alignment for Training-efficient Diffusion Models
Fudan University(复旦大学);Shanghai Academy of Artificial Intelligence for Science(上海科学人工智能研究院);Australian Institute for Machine Learning, The University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
DMAligner: Enhancing Image Alignment via Diffusion Model Based View Synthesis
University of Electronic Science and Technology of China(电子科技大学);Qianyuan Laboratory(钱元实验室);Southwest Jiaotong University(西南交通大学);Independent Researcher(独立研究者)
Conditional Diffusion Posterior Alignment for Sparse-View CT Reconstruction
Swiss Data Science Center (SDSC) in Paul Scherrer Institute (PSI)(瑞士数据科学中心(SDSC)在保罗·舍勒研究所(PSI));Swiss Data Science Center (SDSC) and ETH Zurich(瑞士数据科学中心(SDSC)和苏黎世联邦理工学院);Swiss Data Science Center (SDSC) and Paul Scherrer Institute (PSI)(瑞士数据科学中心(SDSC)和保罗·舍勒研究所(PSI))
ELBO-T2IAlign: A Generic ELBO-Based Method for Calibrating Pixel-level Text-Image Alignment in Diffusion Models
School of Software, Beihang University(北航软件学院);College of Computer and Cyber Security, Hebei Normal University(河北师范大学计算机与网络安全学院);Department of Computer Science, The University of Hong Kong(香港大学计算机科学系)
Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
Harbin Institute of Technology(哈尔滨工业大学);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI));Hong Kong University of Science and Technology(香港科技大学)
Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributes
Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);Peking University(北京大学);University of Nottingham Ningbo China(诺丁汉大学宁波分校)
Where Concept Erasure Should Occur: Concept-Layer Alignment in Text-to-Video Diffusion Models
The School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China(人工智能与自动化学院,华中科技大学,武汉430074,中国);Department of Computer Science and Engineering, University of Nevada, Reno, NV, USA(计算机科学与工程系,内华达大学里诺分校,内华达州,美国)