FlowQ：面向离线强化学习的能量引导流策略
FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
AI总结：
针对离线强化学习场景，提出基于能量引导流匹配的FlowQ算法，通过近似高斯路径学习条件速度场，在取得有竞争力性能的同时实现训练时间与流采样步数无关。
摘要：
在扩散模型领域，引导采样向期望结果收敛的方法已得到广泛研究，尤其在图像生成、轨迹生成等应用中。然而，在训练阶段融入引导机制的相关研究仍相对不足。本文提出能量引导流匹配方法，这是一种增强流模型训练效果的新方案，可消除推理阶段的引导需求。我们通过将能量引导概率路径近似为高斯路径，学习与流策略对应的条件速度场。对于目标分布由数据与能量函数共同定义的任务（如强化学习任务），学习引导轨迹颇具吸引力。基于扩散的策略凭借其表达能力与捕捉多模态动作分布的能力，近期受到广泛关注。这类策略通常采用加权目标函数优化，或通过对策略采样的动作反向传播梯度进行优化。作为替代方案，我们提出FlowQ——一种基于能量引导流匹配的离线强化学习算法。该方法取得了具有竞争力的性能，且策略训练时间不随流采样步数增加而增长。
英文摘要：
The use of guidance to steer sampling toward desired outcomes has been widely explored within diffusion models, especially in applications such as image and trajectory generation. However, incorporating guidance during training remains relatively underexplored. In this work, we introduce energy-guided flow matching, a novel approach that enhances the training of flow models and eliminates the need for guidance at inference time. We learn a conditional velocity field corresponding to the flow policy by approximating an energy-guided probability path as a Gaussian path. Learning guided trajectories is appealing for tasks where the target distribution is defined by a combination of data and an energy function, as in reinforcement learning. Diffusion-based policies have recently attracted attention for their expressive power and ability to capture multi-modal action distributions. Typically, these policies are optimized using weighted objectives or by back-propagating gradients through actions sampled by the policy. As an alternative, we propose FlowQ, an offline reinforcement learning algorithm based on energy-guided flow matching. Our method achieves competitive performance while the policy training time is constant in the number of flow sampling steps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
