Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models with human preferences, inspiring the development of reward-centric diffusion reinforcement learning (RDRL) to achieve similar alignment and controllability. While diffusion models can generate high-quality outputs, RDRL remains susceptible to reward hacking, where the reward score increases without corresponding improvements in perceptual quality. We demonstrate that this vulnerability arises from the non-robustness of reward model gradients, particularly when the reward landscape with respect to the input image is sharp. To mitigate this issue, we introduce methods that exploit gradients from a robustified reward model without requiring its retraining. Specifically, we employ gradients from a flattened reward model, obtained through parameter perturbations of the diffusion model and perturbations of its generated samples. Empirically, each method independently alleviates reward hacking and improves robustness, while their joint use amplifies these benefits. Our resulting framework, RSA-FT (Reward Sharpness-Aware Fine-Tuning), is simple, broadly compatible, and consistently enhances the reliability of RDRL.
Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design
Texas A&M University(德克萨斯A&M大学);EvolutionaryScale(进化尺度);Seoul National University(首尔国立大学);Princeton University(普林斯顿大学);Genentech(基因泰克);University of Texas Health Science Center at Houston(德克萨斯大学健康科学中心休斯顿分校)
IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Models
Applied AI Institute(应用人工智能研究所);MBZUAI(穆罕默德·本·扎耶德人工智能大学);CMAP, École Polytechnique(巴黎综合理工学院应用数学中心);Institute of Foundation Models(基础模型研究所);LRE, EPITA(EPITA 研究与工程实验室)
InstructVideo: Instructing Video Diffusion Models with Human Feedback
Zhejiang University(浙江大学);Alibaba Group(阿里巴巴集团);Tsinghua University(清华大学);Singapore University of Technology and Design(新加坡科技设计大学);Nanyang Technological University(南洋理工大学);University of Cambridge(剑桥大学)
A2D2: Fine-Tuning Any-Length Discrete Diffusion for Adaptive Decoding
Department of Computer and Information Science, University of Pennsylvania(宾夕法尼亚大学计算机与信息科学系);Department of Bioengineering, University of Pennsylvania(宾夕法尼亚大学生物工程系);School of Mathematics, Georgia Institute of Technology(佐治亚理工学院数学学院)
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University(图像处理与模式识别研究所,自动化与智能感知学院,上海交通大学);MoE Key Laboratory of System Control and Information Processing (Shanghai)(系统控制与信息处理MOE重点实验室(上海))
Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Department of Automation, Tsinghua University(清华大学自动化系);Parametrix Technology Company Ltd.(参数科技有限公司)
GRAM: A Generative Foundation Reward Model for Reward Generalization
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院);NiuTrans Research, Shenyang, China(NiuTrans研究);CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS, Beijing, China(中国科学院行为科学重点实验室);Meituan Inc.(美团公司)
Southern University of Science and Technology(南方科技大学);University of Technology Sydney(悉尼科技大学);RIKEN Center for Advanced Intelligence Project (AIP)(理化学研究所先进智能项目中心);Center for Frontier AI Research, Agency for Science Technology(新加坡科技研究局前沿人工智能研究中心);Institute of High Performance Computing, Agency for Science Technology(新加坡科技研究局高性能计算研究所);Nanyang Technological University(南洋理工大学)