arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.26013cs.LGcs.AIcs.CV
收藏

AdvantageFlow: 流模型中基于优势加权的强化学习最小二乘法

AdvantageFlow: Regularized Advantage-Weighted RL in Flow Models

Adobe Research(Adobe研究)

Branislav Kveton, Anup Rao, Subhojyoti Mukherjee, Krishna Kumar Singh, Viet Dac Lai

AI总结:

提出AdvantageFlow算法,通过优势加权前向过程预测损失和 rollout 策略正则化,在图像生成任务中优于Flow-GRPO和负感知微调基线。

摘要:

我们引入了AdvantageFlow,一种用于修正流模型的前向过程强化学习算法。与优化反向过程的Flow-GRPO不同,我们优化了一个优势加权的前向过程预测损失。当优势为负且损失变为非凸时,该优化问题不稳定。我们通过rollout策略正则化来稳定它,这降低了方差,并源于拟合局部奖励改进的目标分布。我们在Stable Diffusion 3.5 Medium上评估了AdvantageFlow在图像生成任务中的表现。它优于Flow-GRPO和基于负感知微调的最先进前向过程强化学习基线。

英文摘要:

We present AdvantageFlow, a forward-process reinforcement learning (RL) algorithm for rectified flow models. The algorithm minimizes an advantage-weighted prediction loss, which maximizes reward, regularized by the rollout policy, which convexifies the objective and makes its optimization stable. Our objective can be viewed as fitting a local reward-improving target distribution. The rollout regularization arises as a variance reduction step. We evaluate AdvantageFlow empirically on text-to-image generation with Stable Diffusion 3.5 Medium and FLUX.1, and compare it to both forward- and reverse-process RL algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑