ExploreNet: 学习在扩散GRPO中何处探索
ExploreNet: Learning Where to Explore in Diffusion GRPO
浏览论文内容
中文总结 AI 辅助
ExploreNet通过自适应预测每个潜元素噪声尺度，在扩散GRPO中实现可学习探索，较Flow-GRPO在GenEval2提升14%，并达到67.2%人类偏好胜率。
中文摘要 AI 辅助
诸如Flow-GRPO等组相对强化学习方法通过在每次去噪步骤中添加各向同性高斯噪声来对图像生成器进行后训练。这种噪声决定了模型从哪些展开轨迹中学习，但它对潜变量的每个通道和空间位置施加相同的扰动。在本文中，我们反而表明潜变量元素在改变生成图像的程度方面存在差异，因此探索应适应这些差异。我们引入EXPLORENET来学习自适应探索分布。EXPLORENET是一个策略，它在观察到任何奖励之前，根据当前潜变量、去噪步骤和提示预测每个潜变量元素的噪声尺度；它基于每个展开组的奖励离散度进行训练，并在训练后被丢弃，推理过程保持不变。在Stable Diffusion 3.5 Medium上，EXPLORENET在保留的GenEval2上比Flow-GRPO提高了14%，可迁移到两个独立的组合基准和五个偏好及图像质量模型，并达到67.2%的人类偏好胜率。总体而言，在我们的组相对扩散强化学习实验中，我们发现探索是可学习的，探索分布的形状比其幅度更重要，展开轨迹的质量比数量更有效。
英文摘要
Group-relative RL methods such as Flow-GRPO post-train image generators by exploring with isotropic Gaussian noise added at every denoising step. This noise decides which rollouts the model learns from, yet it perturbs every channel and spatial position of the latent equally. In this paper, we instead show that latent elements differ in how much they change the generated image, so exploration should adapt to these differences. We introduce EXPLORENET to learn an adaptive exploration distribution. EXPLORENET is a policy that predicts a noise scale for every latent element from the current latent, the denoising step, and the prompt, before any reward is observed; it is trained on the reward spread of each rollout group and discarded after training, leaving inference unchanged. On Stable Diffusion 3.5 Medium, EXPLORENET improves held-out GenEval2 by 14% over Flow-GRPO, transfers to two independent compositional benchmarks and five preference and image-quality models, and reaches a 67.2% human preference win-rate. Overall, across our group-relative diffusion RL experiments, we find that exploration is learnable, the shape of the exploration distribution outweighs its magnitude, and rollout quality is more effective than rollout quantity.
发表机构
- University of Washington（华盛顿大学）
机构由 AI 辅助整理，请以论文原文为准。
