arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.21175cs.LGcs.AI
收藏

奖励敏锐度感知的扩散模型微调

Reward Sharpness-Aware Fine-Tuning for Diffusion Models

Department of AI Convergence, GIST(人工智能融合系,韩国科学技术院);Samsung Research(三星研究院)

Kwanyoung Kim, Byeongsu Sim

AI总结:

本文提出RSA-FT方法,通过改进奖励模型梯度的鲁棒性来缓解扩散模型中的奖励黑客问题,提升RDRL的可靠性。

摘要:

本文提出RSA-FT方法,通过改进奖励模型梯度的鲁棒性来缓解扩散模型中的奖励黑客问题,提升RDRL的可靠性。

英文摘要:

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models with human preferences, inspiring the development of reward-centric diffusion reinforcement learning (RDRL) to achieve similar alignment and controllability. While diffusion models can generate high-quality outputs, RDRL remains susceptible to reward hacking, where the reward score increases without corresponding improvements in perceptual quality. We demonstrate that this vulnerability arises from the non-robustness of reward model gradients, particularly when the reward landscape with respect to the input image is sharp. To mitigate this issue, we introduce methods that exploit gradients from a robustified reward model without requiring its retraining. Specifically, we employ gradients from a flattened reward model, obtained through parameter perturbations of the diffusion model and perturbations of its generated samples. Empirically, each method independently alleviates reward hacking and improves robustness, while their joint use amplifies these benefits. Our resulting framework, RSA-FT (Reward Sharpness-Aware Fine-Tuning), is simple, broadly compatible, and consistently enhances the reliability of RDRL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑