arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.09159cs.LG
收藏

截断修正流策略用于具有一步采样的强化学习

Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling

Harbin Institute of Technology(哈尔滨工业大学)

Xubin Zhou, Yipeng Yang, Zhan Li

AI总结:

本文提出TRFP框架,通过混合确定性-随机性架构解决MaxEnt RL中连续时间生成策略的可微性问题,实现稳定训练和有效一步采样,实验表明其在多目标环境和MuJoCo基准中表现优异。

摘要:

最大熵强化学习(MaxEnt RL)已成为序列决策制定的标准框架,但其标准高斯策略参数化本质上是单峰的,限制了其对复杂多模态动作分布的建模能力。这种限制推动了基于扩散和流匹配的生成策略的兴趣,但将其整合到MaxEnt RL中存在两大挑战:连续时间生成策略的似然和熵通常不可微,且多步采样引入了长周期反向传播不稳定性和显著的推理延迟。为了解决这些挑战,我们提出了截断修正流策略(TRFP),其基于混合确定性-随机性架构。这种设计使熵正则化优化成为可能,同时通过梯度截断和流平滑支持稳定训练和有效的一步采样。在玩具多目标环境和10个MuJoCo基准上的实验证明,TRFP能够有效捕捉多模态行为,在标准采样下优于大多数基线,在一步采样下仍保持高度竞争力。

英文摘要:

Maximum entropy reinforcement learning (MaxEnt RL) has become a standard framework for sequential decision making, yet its standard Gaussian policy parameterization is inherently unimodal, limiting its ability to model complex multimodal action distributions. This limitation has motivated increasing interest in generative policies based on diffusion and flow matching as more expressive alternatives. However, incorporating such policies into MaxEnt RL is challenging for two main reasons: the likelihood and entropy of continuous-time generative policies are generally intractable, and multi-step sampling introduces both long-horizon backpropagation instability and substantial inference latency. To address these challenges, we propose Truncated Rectified Flow Policy (TRFP), a framework built on a hybrid deterministic-stochastic architecture. This design makes entropy-regularized optimization tractable while supporting stable training and effective one-step sampling through gradient truncation and flow straightening. Empirical results on a toy multigoal environment and 10 MuJoCo benchmarks show that TRFP captures multimodal behavior effectively, outperforms strong baselines on most benchmarks under standard sampling, and remains highly competitive under one-step sampling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑