arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01591cs.CV

Know Your Step: 通过步感知优势实现更快速和更优的流匹配模型对齐

Faster and Better Alignment for Flow Matching Models via Step-aware Advantages

Zhejiang University(浙江大学);HiThink Research(HiThink研究);Southern University of Science and Technology(南方科技大学)

Zhixiong Yue, Feiyang Ye, Zixuan Ni, Sheng Shen, Yu Zhang

AI总结:

TAFS GRPO通过步感知优势机制改进流匹配模型,实现更高效且更优的少步文本到图像生成与人类偏好对齐。

摘要:

近期流匹配模型在强化学习(RL)方面的进展显著提升了少步文本到图像生成中的人类偏好对齐。然而,现有基于RL的流匹配模型方法通常依赖大量的去噪步骤,而面临稀疏且不精确的奖励信号,往往导致次优对齐。为了解决这些限制,我们提出温度退火少步采样与组相对策略优化(TAFS GRPO),一种新的框架用于训练流匹配文本到图像模型成为高效的少步生成器,与人类偏好良好对齐。我们的方法迭代地将自适应时间噪声注入一步样本的结果中。通过反复退火模型的采样输出,它在采样过程中引入了随机性,同时保持每个生成图像的语义完整性。此外,其步感知优势整合机制结合GRPO以避免奖励函数可微的需求,并提供密集且步特定的奖励以稳定策略优化。广泛的实验表明,TAFS GRPO在少步文本到图像生成中实现了强大的性能,并显著提高了生成图像与人类偏好的对齐程度。本工作的代码和模型将向公众开放,以促进进一步研究。

英文摘要:

Recent advances in flow matching models, particularly with reinforcement learning (RL), have significantly enhanced human preference alignment in few-step text-to-image generators. However, existing RL-based approaches for flow matching models typically rely on numerous denoising steps, while suffering from sparse and imprecise reward signals that often lead to suboptimal alignment. To address these limitations, we propose Temperature-Annealed Few-step Sampling with Group Relative Policy Optimization (TAFS-GRPO), a novel framework for training flow matching text-to-image models into efficient few-step generators well aligned with human preferences. Our method iteratively injects adaptive time-dependent noise into one-step clean predictions. By repeatedly annealing the model's sampled outputs, it introduces stochasticity into the sampling process while preserving the semantic integrity of each generated image. Moreover, its step-aware advantage integration mechanism combines GRPO with temperature-annealed sampling to eliminate the need for a differentiable reward function and provide dense, step-specific rewards for stable policy optimization. Extensive experiments demonstrate that TAFS-GRPO achieves strong performance in few-step text-to-image generation and significantly improves the alignment of generated images with human preferences. The code and models of this work will be available to facilitate further research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑