通过强化学习和奖励建模对扩散模型的对齐与安全性：综述
Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
AI总结：
本文综述了通过强化学习、奖励建模等方法对文本到图像扩散模型进行对齐和安全性的最新进展，探讨了反馈来源、奖励信号形式、优化机制等五个维度，并提出了多目标对齐、反馈高效偏好学习等开放性挑战。
摘要：
扩散模型已成为图像和多模态生成的核心范式，但其部署引发了关于对齐、安全性、偏好满足和滥用鲁棒性的持续疑问。本文综述了通过强化学习、奖励建模、偏好优化和安全特定微调对齐文本到图像扩散模型的最新进展。我们沿五个轴组织文献：反馈来源、奖励或偏好信号形式、优化机制、分布偏移和奖励过度优化的处理、以及安全作为显式约束而非一般偏好的程度。本文涵盖了人类反馈强化学习、KL正则化策略优化、直接偏好优化、二元效用优化、可微奖励微调、替代奖励学习、区域感知微调以及安全导向的DPO变体。为使综述易于理解，我们包含了扩散采样、奖励建模和偏好优化的教程解释，并简要连接了图像扩散对齐与新兴文本和掩码语言扩散模型。我们还比较了代表性方法在反馈要求、计算成本、可扩展性、对奖励黑客的易感性和安全性关键部署的适用性。最后，我们将文献综合为一组开放性挑战：多目标对齐、反馈高效偏好学习、对抗鲁棒安全对齐、在变化规范下的持续对齐和可解释奖励建模。本文的目标是为新兴的扩散模型对齐领域提供一个连贯的技术图谱，并识别在对齐生成模型可靠部署前必须解决的方法学缺口。
英文摘要：
Diffusion models have become a central paradigm for image and multimodal generation, yet their deployment raises persistent questions about alignment, safety, preference satisfaction, and robustness to misuse. This survey reviews recent progress on aligning text-to-image diffusion models through reinforcement learning, reward modeling, preference optimization, and safety-specific fine-tuning. We organize the literature along five axes: the source of feedback, the form of the reward or preference signal, the optimization mechanism, the treatment of distribution shift and reward overoptimization, and the extent to which safety is addressed as an explicit constraint rather than a generic preference. The review covers reinforcement learning from human feedback, KL-regularized policy optimization, direct preference optimization, binary utility optimization, differentiable reward fine-tuning, surrogate reward learning, region-aware fine-tuning, and safety-oriented DPO variants. To make the survey accessible, we include tutorial explanations of diffusion sampling, reward modeling, and preference optimization, and briefly connect image diffusion alignment to emerging text and masked language diffusion models. We also compare representative methods in terms of feedback requirements, computational cost, scalability, susceptibility to reward hacking, and suitability for safety-critical deployment. Finally, we synthesize the literature into a set of open challenges: multi-objective alignment, feedback-efficient preference learning, adversarially robust safety alignment, continual alignment under changing norms, and interpretable reward modeling. The goal of this survey is to provide a coherent technical map of the emerging area of diffusion model alignment and to identify the methodological gaps that must be addressed before aligned generative models can be reliably deployed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
