Diffusion models have achieved remarkable results in image generation, and have similarly been used to learn high-performing policies in sequential decision-making tasks. Decision-making diffusion models can be trained on lower-quality data, and then be steered with a reward function to generate near-optimal trajectories. We consider the problem of extracting a reward function by comparing a decision-making diffusion model that models low-reward behavior and one that models high-reward behavior; a setting related to inverse reinforcement learning. We first define the notion of a relative reward function of two diffusion models and show conditions under which it exists and is unique. We then devise a practical learning algorithm for extracting it by aligning the gradients of a reward function -- parametrized by a neural network -- to the difference in outputs of both diffusion models. Our method finds correct reward functions in navigation environments, and we demonstrate that steering the base model with the learned reward functions results in significantly increased performance in standard locomotion benchmarks. Finally, we demonstrate that our approach generalizes beyond sequential decision-making by learning a reward-like function from two large-scale image generation diffusion models. The extracted reward function successfully assigns lower rewards to harmful images.
Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)科学与工程学院);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)
Inference-Time Alignment of Diffusion Models with Direct Noise Optimization
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));DAMO Academy, Alibaba Group(阿里巴巴达摩院);Hupan Lab, Zhejiang Province(浙江省湖畔实验室);University of Minnesota, USA(美国明尼苏达大学);Shenzhen Research Institute of Big Data(深圳大数据研究院)
Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Department of Automation, Tsinghua University(清华大学自动化系);Parametrix Technology Company Ltd.(参数科技有限公司)
Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design
Texas A&M University(德克萨斯A&M大学);EvolutionaryScale(进化尺度);Seoul National University(首尔国立大学);Princeton University(普林斯顿大学);Genentech(基因泰克);University of Texas Health Science Center at Houston(德克萨斯大学健康科学中心休斯顿分校)
NVIDIA(NVIDIA公司);University of Toronto(多伦多大学);Vector Institute(向量研究所);Harvard University(哈佛大学);Kempner Institute(凯姆纳研究所);IAIFI(IAIFI机构);Microsoft Research(微软研究院);Carnegie Mellon University(卡内基梅隆大学);Courant Institute, New York University(纽约大学应用数学学院);ML Lab at Capital Fund Management (CFM)(Capital Fund Management (CFM)机器学习实验室)
Exploratory Diffusion Model for Unsupervised Reinforcement Learning
Department of Computer Science & Technology, Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University(计算机科学与技术系、人工智能研究院、BNRist中心、清华-博世联合机器学习中心、THBI实验室、清华大学)
IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Models
Applied AI Institute(应用人工智能研究所);MBZUAI(穆罕默德·本·扎耶德人工智能大学);CMAP, École Polytechnique(巴黎综合理工学院应用数学中心);Institute of Foundation Models(基础模型研究所);LRE, EPITA(EPITA 研究与工程实验室)
Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases
Institute of Artificial Intelligence, School of Computer Science, Wuhan University, China;Hubei Luojia Laboratory, Wuhan, China;The University of Sydney, Australia;JD Explore Academy, Beijing, China;Nanyang Technological University, Singapore
TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
InstructVideo: Instructing Video Diffusion Models with Human Feedback
Zhejiang University(浙江大学);Alibaba Group(阿里巴巴集团);Tsinghua University(清华大学);Singapore University of Technology and Design(新加坡科技设计大学);Nanyang Technological University(南洋理工大学);University of Cambridge(剑桥大学)
TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment
Huazhong University of Science and Technology(华中科技大学);Kuaishou Technology(快手科技);Nanyang Technological University(南洋理工大学);Wuhan University(武汉大学);Tsinghua University(清华大学)