arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.41949cs.AIcs.LG
收藏

优化用于少步流强化学习的随机策略几何

Optimizing Stochastic Policy Geometry for Few-Step Flow Reinforcement Learning

Beihang University(北京航空航天大学);Sensetime Research(商汤科技研究院)

Zhuguanyu Wu, Congjia Chen, Ruihao Gong, Yang Yong, Shiqiao Gu, Yi Xu, Xianglong Liu

AI总结:

本研究提出Geo-FSRL方法,通过优化随机策略几何提升少步流强化学习性能,在4步设置下图像与视频生成任务中优于Flow-GRPO和Flow-CPS。

摘要:

少步流模型通过少量大规模去噪转换生成样本。将GRPO应用于此类模型需要在去噪转换中引入随机性,当单个转换跨越大范围噪声水平时,所得随机策略的选择成为优化的关键因素。近期的构造方法约束策略以保留目标流系数,但该约束允许一组策略,且未提供从中选择的准则。我们提出Geo-FSRL(几何优化少步强化学习),其基于选择原理:在固定条件状态和时间步对下,选择使任意给定非零速度输出更新所诱导的条件KL散度最小的容许策略。在保系数的各向同性高斯转换族中,该准则简化为最小化单个标量几何因子,该因子决定了每个速度输出更新的条件KL。同一因子还控制速度空间费舍尔信息和条件转换对数似然比方差。我们推导了该族内的唯一闭式优化器,其适用于所有非零更新幅度和方向,转换系数由源噪声水平和目标噪声水平共同决定。由于最小化的几何因子仍随时间步变化,我们进一步采用先前工作的梯度尺度匹配,推导了闭式策略梯度和KL权重,以归一化显式得分尺度和参考KL曲率。Geo-FSRL在原始少步调度上训练,并在推理时保留确定性求解器。在4步设置下,我们的方法在图像生成和视频生成模型上均取得比Flow-GRPO和Flow-CPS更高的最终奖励分数。代码可在该https URL获取。

英文摘要:

Few-step flow models generate samples through a handful of large denoising transitions. Applying GRPO to such models requires introducing stochasticity into the denoising transitions, and when a single transition spans a wide range of noise levels, the choice of the resulting stochastic policy becomes a major factor in optimization. Recent constructions constrain the policy to preserve the target flow coefficients, but this constraint admits a family of policies and offers no criterion for choosing among them. We introduce Geo-FSRL (Geometry-Optimized Few-Step Reinforcement Learning), built on a selection principle: at a fixed conditioning state and timestep pair, choose the admissible policy that minimizes the conditional KL divergence induced by any given nonzero velocity-output update. Within a coefficient-preserving family of isotropic Gaussian transitions, this criterion reduces to minimizing a single scalar geometry factor that determines the conditional KL for every velocity-output update. The same factor governs velocity-space Fisher information and conditional transition log-likelihood-ratio variance. We derive the unique closed-form optimizer within this family, shared across all nonzero update magnitudes and directions, with transition coefficients determined by both the source and target noise levels. Because the minimized geometry factor still varies across timesteps, we further adopt gradient-scale matching from prior work and derive closed-form policy-gradient and KL weights that normalize explicit score scales and reference-KL curvature. Geo-FSRL trains on the original few-step schedule and retains the deterministic solver at inference. Our method achieves higher final reward scores than Flow-GRPO and Flow-CPS on both image generation and video generation models under the 4-step setting. Code is available at https://github.com/ModelTC/LightX2V/tree/main/lightx2v_rl

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑