PromptRL: 流基于图像生成中强化学习中提示的重要性
PromptRL: Prompt Matters in RL for Flow-Based Image Generation
AI总结:
PromptRL通过整合语言模型作为可训练的提示精修代理,提升流基于图像生成中强化学习的性能和效率。
摘要:
流匹配模型(FMs)已经彻底改变了文本到图像(T2I)生成,强化学习(RL)作为关键的后训练策略,用于与奖励目标对齐。在本研究中,我们表明,当前FMs的RL流程存在两个未被充分认识但重要的限制:由于生成多样性不足导致的样本效率低下,以及明显的提示过拟合现象,即模型记忆特定的训练公式,并在评估时在语义等价但风格不同的提示下表现出剧烈性能崩溃。我们提出了PromptRL(流基于图像生成中强化学习中提示的重要性),一个框架,将语言模型(LMs)作为可训练的提示精修代理直接整合到基于流的RL优化循环中。这种设计带来了两个互补的好处:快速开发复杂的提示重写能力,以及关键的协同训练制度,重塑优化动态。PromptRL在多个基准上实现了最先进的性能,获得了GenEval 0.97、OCR准确度 0.98 和 PickScore 24.05 的分数。此外,我们验证了我们的RL方法在大规模图像编辑模型上的有效性,通过仅 0.06 百万次回滚,将FLUX.1-Kontext的EditReward从1.19提升到1.43,超过了Gemini 2.5 Flash Image(也称为Nano Banana),其得分为1.37,并实现了与ReasonNet(1.44)相当的性能,后者依赖于细粒度的数据注释和复杂的多阶段训练。我们的广泛实验表明,PromptRL在要求的回滚次数上比朴素的流-only RL少超过2倍,同时一致地达到了更高的性能上限。我们的代码可在https://github.com/G-U-N/UniRL上获得。
英文摘要:
Flow matching models (FMs) have revolutionized text-to-image (T2I) generation, with reinforcement learning (RL) serving as a critical post-training strategy for alignment with reward objectives. In this research, we show that current RL pipelines for FMs suffer from two underappreciated yet important limitations: sample inefficiency due to insufficient generation diversity, and pronounced prompt overfitting, where models memorize specific training formulations and exhibit dramatic performance collapse when evaluated on semantically equivalent but stylistically varied prompts. We present PromptRL (Prompt Matters in RL for Flow-Based Image Generation), a framework that incorporates language models (LMs) as trainable prompt refinement agents directly within the flow-based RL optimization loop. This design yields two complementary benefits: rapid development of sophisticated prompt rewriting capabilities and, critically, a synergistic training regime that reshapes the optimization dynamics. PromptRL achieves state-of-the-art performance across multiple benchmarks, obtaining scores of 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore. Furthermore, we validate the effectiveness of our RL approach on large-scale image editing models, improving the EditReward of FLUX.1-Kontext from 1.19 to 1.43 with only 0.06 million rollouts, surpassing Gemini 2.5 Flash Image (also known as Nano Banana), which scores 1.37, and achieving comparable performance with ReasonNet (1.44), which relied on fine-grained data annotations along with a complex multi-stage training. Our extensive experiments empirically demonstrate that PromptRL consistently achieves higher performance ceilings while requiring over 2$\times$ fewer rollouts compared to naive flow-only RL. Our code is available at https://github.com/G-U-N/UniRL.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。