Drawing upon recent advances in language model alignment, we formulate offline Reinforcement Learning as a two-stage optimization problem: First pretraining expressive generative policies on reward-free behavior datasets, then fine-tuning these policies to align with task-specific annotations like Q-values. This strategy allows us to leverage abundant and diverse behavior data to enhance generalization and enable rapid adaptation to downstream tasks using minimal annotations. In particular, we introduce Efficient Diffusion Alignment (EDA) for solving continuous control problems. EDA utilizes diffusion models for behavior modeling. However, unlike previous approaches, we represent diffusion policies as the derivative of a scalar neural network with respect to action inputs. This representation is critical because it enables direct density calculation for diffusion models, making them compatible with existing LLM alignment theories. During policy fine-tuning, we extend preference-based alignment methods like Direct Preference Optimization (DPO) to align diffusion behaviors with continuous Q-functions. Our evaluation on the D4RL benchmark shows that EDA exceeds all baseline methods in overall performance. Notably, EDA maintains about 95\% of performance and still outperforms several baselines given only 1\% of Q-labelled data during fine-tuning.
Forward KL Regularized Preference Optimization for Aligning Diffusion Policies
Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；Tsinghua University（清华大学）；Northwestern Polytechnical University Xi’an（西北工业大学（西安））；Hong Kong University of Science and Technology（香港科技大学）；ShenZhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）
SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models
Shanghai Science and Intelligence Institute, Shanghai, China（上海科学与智能研究所）；Fudan University, Shanghai, China（复旦大学）；Australian Institute for Machine Learning, The University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）
Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers
School of Computer Science, Peking University（北京大学计算机科学学院）；School of Software Engineering, University of Electronic Science and Technology of China（电子科技大学软件工程学院）
Aligning Few-Step Diffusion Models with Dense Reward Difference Learning
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（计算机学院、多媒体软件国家工程研究中心和多媒体与网络通信工程湖北省重点实验室、武汉大学）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（网络安全科学与技术学院、中山大学深圳校区）；TikTok, ByteDance（TikTok、字节跳动）；Tencent Inc.（腾讯公司）；College of Electronic and Information Engineering, Tongji University（电子信息工程学院、同济大学）；School of Medical Information and Engineering, Southwest Medical University（医学信息与工程学院、西南医科大学）；College of Computing and Data Science and the Generative AI Lab at Nanyang Technological University（计算与数据科学学院和南洋理工大学生成式AI实验室）
Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generation
Massachusetts Institute of Technology（麻省理工学院）；Technical University of Denmark（丹麦技术大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；University of Copenhagen（哥本哈根大学）
本文提出 DOM 与 Trajectory Alignment 框架，通过对齐扩散采样轨迹和物理优化轨迹，在结构拓扑优化中实现少步、低成本且可行的高性能设计生成。
Stochastic Clock Attention for Aligning Continuous and Ordered Sequences
Department of Physics Education, Seoul National University, Seoul 08826, Korea（物理教育系，首尔国立大学）；Center for Theoretical Physics and Artificial Intelligence Institute, Seoul National University, Seoul 08826, Korea（理论物理与人工智能研究所，首尔国立大学）；Artificial Intelligence Institute, Seoul National University, Seoul 08826, Korea（人工智能研究所，首尔国立大学）；School of Computational Sciences, Korea Institute for Advanced Study, Seoul 02455, Korea（计算科学学院，韩国高级研究院）
Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
University of Liverpool（利物浦大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Edinburgh（爱丁堡大学）；University of Bristol（布里斯托大学）；University of Exeter（埃克塞特大学）
Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes
School of Computer Science and Technology, University of Science and Technology of China（计算机科学与技术学院，中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）
CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
LUMIA Lab（LUMIA实验室）；School of Artificial Intelligence（人工智能学院）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；School of Computer Science（计算机科学学院）；Shanghai AI Laboratory（上海人工智能实验室）
