Group Relative Policy Optimization has emerged as essential for aligning video diffusion models with human preferences, but faces a critical computational bottleneck: training a 14B parametered model typically demands hundreds of GPU days per experiment. Existing efficiency methods reduce costs through sliding window subsampling training timesteps, but fundamentally compromise optimization, exhibiting severe instability and failing to reach full trajectory performance. We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets while substantially improving training efficiency. Flash-GRPO addresses two critical challenges: iso-temporal grouping eliminates timestep-confounded variance by enforcing prompt-wise temporal consistency, decoupling policy performance from timestep difficulty; temporal gradient rectification neutralizes the time-dependent scaling factor that causes vastly inconsistent gradient magnitudes across timesteps. Experiments on 1.3B to 14B parameter models validate Flash-GRPO's effectiveness, demonstrating substantial training acceleration with consistent stability and state-of-the-art alignment quality.
Rethinking Reward Signals in Video GRPO: When Scores Become Targets
University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信)
MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
Hunyuan, Tencent(腾讯混元);School of Computer Science, Peking University(北京大学计算机学院);Computer Center, Peking University(北京大学计算中心);Department of Engineering Physics, Tsinghua University(清华大学工程物理系)
AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO
Shanghai Jiao Tong University(上海交通大学);S-Lab, Nanyang Technological University(南洋理工大学S实验室);Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Stanford University(斯坦福大学);Shanghai Innovation Institute(上海创新研究院);The Chinese University of Hong Kong(香港中文大学);Fudan University(复旦大学);CPII under InnoHK(InnoHK下的CPII);Adobe Research(Adobe研究)
Seeing What Matters: Visual Preference Policy Optimization for Visual Generation
Southeast University(东南大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);University of Science and Technology of China(中国科学技术大学)
GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University(人工智能学院(iOPEN),西北工业大学);HuaWei Technologies Co., Ltd.(华为技术有限公司);The University of Hong Kong(香港大学)
VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization
Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));Alibaba International Group(阿里巴巴国际集团);Division of AMC and Department of ECE, HKUST(HKUST电子工程系与AMC division)