扩散模型上的像素级RL：来自丰富反馈的强化学习
Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback
AI总结：
针对DDPO仅有最终图像奖励导致的稀疏奖励与高样本需求，本文提出PXPO算法，将像素级反馈纳入扩散模型强化学习，以更精细的奖励优化策略。
摘要：
潜在扩散模型是合成图像生成领域的最先进方法。为了使这些模型与人类偏好保持一致，利用基于人类反馈的强化学习来训练模型至关重要。Black等人于2024年提出了去噪扩散策略优化（DDPO），该方法通过将生成过程建模为带有最终奖励的马尔可夫链，来解释生成过程的迭代去噪特性。由于奖励只是决定模型在整张图像上表现的单一数值，模型必须在极其稀疏的奖励环境中进行探索，因此需要大量样本。在本研究中，我们扩展了DDPO，提出了像素级策略优化（PXPO）算法，该算法能够接收每个像素的反馈，从而为模型提供更精细的奖励。
英文摘要：
Latent diffusion models are the state-of-the-art for synthetic image generation. To align these models with human preferences, training the models using reinforcement learning on human feedback is crucial. Black et. al 2024 introduced denoising diffusion policy optimisation (DDPO), which accounts for the iterative denoising nature of the generation by modelling it as a Markov chain with a final reward. As the reward is a single value that determines the model's performance on the entire image, the model has to navigate a very sparse reward landscape and so requires a large sample count. In this work, we extend the DDPO by presenting the Pixel-wise Policy Optimisation (PXPO) algorithm, which can take feedback for each pixel, providing a more nuanced reward to the model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
