在线扩散强化学习的路由策略优化
Routing Policy Optimization for Online Diffusion Reinforcement Learning
AI总结：
本文提出RPO方法，将注意力路由转为随机探索动作，采用受控双回滚估计器，在保留生成器原始采样器的同时提升生成质量，训练效率较Flow-GRPO类方法高近4倍。
摘要：
强化学习可使视觉生成器对齐预训练目标之外的奖励信号，但现有基于得分函数的方法通常通过扰动反向去噪轨迹引入随机性，导致基于回滚的优化成本高昂。本文探究能否在保留生成器原始采样器的同时，在生成器内部引入随机探索。路由策略优化（Routing Policy Optimization, RPO）解决了该问题，将注意力路由转化为随机探索动作。在少数选定的去噪层级，每个查询块保留自身块，并按生成器原生注意力质量比例采样其他关键块，或保留密集注意力。采样的路由会扰动该转换的速度，围绕密集速度的高斯似然以得分函数目标对扰动进行评分。RPO进一步采用受控双回滚估计器：两次回滚共享相同的提示、初始潜变量、路由去噪层级和密集锚定层，仅采样的注意力路由不同，其奖励差为路由转换提供优势信号。由于似然可对密集速度本身求导，移除路由后学习到的更新仍有效。推理时，RPO恢复密集注意力并保留原始确定性采样器。在文本到图像和文本到视频生成实验中，RPO提升了生成质量，且训练效率比Flow-GRPO类方法高近4倍。更多视觉结果请访问项目页面：this https URL
英文摘要：
Reinforcement learning can align visual generators with reward signals beyond their pretraining objectives, but existing score-function methods typically introduce stochasticity by perturbing the reverse denoising trajectory, leading to costly rollout-based optimization. We instead ask whether stochastic exploration can be introduced inside the generator while preserving its original sampler. Routing Policy Optimization (RPO) answers this question by turning attention routing into a stochastic exploration action. At a few selected denoising levels, each query block keeps its own block and samples further key blocks in proportion to the generator's native attention mass, or keeps dense attention. The sampled route perturbs the velocity of that transition, and a Gaussian likelihood around the dense velocity scores the perturbation with a score-function objective. RPO further uses a controlled two-rollout estimator: two rollouts share the same prompt, initial latent, routed denoising levels, and dense anchor layers, differing only in their sampled attention routes. Their reward difference provides an advantage signal for the routed transitions. Because the likelihood differentiates the dense velocity itself, the learned updates remain effective after routing is removed. At inference, RPO restores dense attention and retains the original deterministic sampler. Experiments on text-to-image and text-to-video generation show that RPO improves generation quality while achieving nearly 4x higher training efficiency than Flow-GRPO-style methods. Please see our project page at https://velox-world.com/rpo for visual results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
