Post training via GRPO has demonstrated remarkable effectiveness in improving the generation quality of flow-matching models. However, GRPO suffers from inherently low sample efficiency due to its on-policy training paradigm. To address this limitation, we present OP-GRPO, the first Off-Policy GRPO framework tailored for flow-matching models. First, we actively select high-quality trajectories and adaptively incorporate them into a replay buffer for reuse in subsequent training iterations. Second, to mitigate the distribution shift introduced by off-policy samples, we propose a sequence-level importance sampling correction that preserves the integrity of GRPO's clipping mechanism while ensuring stable policy updates. Third, we theoretically and empirically show that late denoising steps yield ill-conditioned off-policy ratios, and mitigate this by truncating trajectories at late steps. Across image and video generation benchmarks, OP-GRPO achieves comparable or superior performance to Flow-GRPO with only 34.2% of the training steps on average, yielding substantial gains in training efficiency while maintaining generation quality.
Rethinking Reward Signals in Video GRPO: When Scores Become Targets
University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信)
AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO
Shanghai Jiao Tong University(上海交通大学);S-Lab, Nanyang Technological University(南洋理工大学S实验室);Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Stanford University(斯坦福大学);Shanghai Innovation Institute(上海创新研究院);The Chinese University of Hong Kong(香港中文大学);Fudan University(复旦大学);CPII under InnoHK(InnoHK下的CPII);Adobe Research(Adobe研究)
Seeing What Matters: Visual Preference Policy Optimization for Visual Generation
Southeast University(东南大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);University of Science and Technology of China(中国科学技术大学)
GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University(人工智能学院(iOPEN),西北工业大学);HuaWei Technologies Co., Ltd.(华为技术有限公司);The University of Hong Kong(香港大学)
VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization
Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));Alibaba International Group(阿里巴巴国际集团);Division of AMC and Department of ECE, HKUST(HKUST电子工程系与AMC division)