We present AdvantageFlow, a forward-process reinforcement learning (RL) algorithm for rectified flow models. The algorithm minimizes an advantage-weighted prediction loss, which maximizes reward, regularized by the rollout policy, which convexifies the objective and makes its optimization stable. Our objective can be viewed as fitting a local reward-improving target distribution. The rollout regularization arises as a variance reduction step. We evaluate AdvantageFlow empirically on text-to-image generation with Stable Diffusion 3.5 Medium and FLUX.1, and compare it to both forward- and reverse-process RL algorithms.
Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
本文研究了基于情境强盗模型的自动决策系统中的离线学习问题。我们得到了由上下文、(随机)动作和(非负)奖励组成的记录历史数据。一个共同的目标是评估如果在相同的上下文中采取不同的行动会发生什么,以便相应地优化行动策略。该问题的典型方法是逆概率加权估计(IPWE)[Bottou等人,2013],需要记录动作概率,这在实践中可能由于工程复杂性而缺失。即使在可用的情况下,小的动作概率也会导致IPWE中的大的不确定性,从而使相应的结果变得微不足道。为了解决这两个问题,我们展示了如何使用政策改进(PIL)的目标,正规化的政策模仿(IML)。我们激励和分析PIL作为一个扩展,剪切IPWE,通过显示,这两个都是下界代理香草IPWE。我们还将IML正式连接到IPWE方差估计[Swaminathan and Joachims 2015]和自然策略梯度。在没有概率记录的情况下,我们的PIL-IML解释通过奖励加权来证明和改进最先进的交叉熵(CE)损失,该损失预测在相同上下文中可用的所有动作候选者中的动作项。与概率日志,我们的主要理论贡献连接IML欠拟合的存在,无论是混杂变量或模型误设定。我们通过基于辛普森悖论、标准UCI多类到强盗转换和Criteo反事实分析挑战数据集的模拟来展示我们见解的价值和准确性。
Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice
The University of Tokyo(东京大学);OMRON SINIC X(欧姆龙 SINIC X 研究院);DeepMind(深度思维);Google Research(谷歌研究院);Peking University(北京大学);Otto von Guericke University Magdeburg(马格德堡奥托·冯·格里克大学);University of Alberta(阿尔伯塔大学)
DeepMpMRI: Tensor-decomposition Regularized Learning for Fast and High-Fidelity Multi-Parametric Microstructural MR Imaging
Paul C. Lauterbur Research Center for Biomedical Imaging(Paul C. Lauterbur生物医学成像研究中心);Shenzhen Institutes of Advanced Technology(深圳先进技术研究院);Chinese Academy of Sciences(中国科学院);State Key Laboratory of Complex & Critical Software Environment(复杂与关键软件环境国家重点实验室);Beihang University(北航);School of Biomedical Engineering(生物医学工程学院);Tsinghua University(清华大学);School of Physics and Electronic Science(物理与电子科学学院);Changsha University of Science and Technology(长沙理工大学);Faculty of Computer Science and Control Engineering(计算机科学与控制工程学院);Shenzhen University of Advanced Technology(深圳大学先进技术学院);Department of Information Engineering(信息工程学院);Zhejiang University of Technology(浙江工业大学)
Worth Their Weight: Randomized and Regularized Block Kaczmarz Algorithms without Preprocessing
Department of Mathematics, UC Berkeley(数学系,加州大学伯克利分校);Applied Mathematics and Computational Research Division, Lawrence Berkeley National Laboratory(应用数学与计算研究部,劳伦斯伯克利国家实验室)
Decentralized Online Regularized Learning Over Random Time-Varying Graphs
East China Normal University(华东师范大学);No.2 High School of East China Normal University(华东师范大学第二附属中学);Institute of Systems Science, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院系统科学研究所);University of Chinese Academy of Sciences(中国科学院大学);Ningbo University(宁波大学)