arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.01427cs.LGcs.RO
收藏

通过整合扩散模型和近端策略优化提升强化学习的样本效率和探索

Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization

Bauman Moscow State Technical University(鲍曼莫斯科国立技术大学)

Tianci Gao, Konstantin A. Neusypin, Dmitry D. Dmitriev, Bo Yang, Shengren Rao

AI总结:

本文提出PPO-DAP方法,通过整合扩散模型和近端策略优化,提升强化学习的样本效率和探索能力,有效改进了连续控制任务中的学习效率和性能。

摘要:

近端策略优化(PPO)因其鲁棒性和稳定的训练过程而广泛应用于连续控制,但其在具有昂贵交互和高维动作空间的任务中仍显得样本效率低下。本文提出PPO-DAP(具有扩散动作先验的PPO),这是一种严格在线策略框架,通过不修改PPO目标来提升探索质量和学习效率。PPO-DAP遵循两阶段协议。离线阶段,我们在记录的轨迹上预训练一个条件扩散动作先验,以覆盖由行为策略支持的动作分布。在线阶段,PPO仅使用新收集的在线轨迹更新actor-critic,同时通过参数高效调优(Adapter/LoRA)在小参数子集上调整扩散先验以适应在线状态分布。对于每个在线状态,先验生成多个动作提案,并利用基于批评机的能量重新加权和去噪梯度指导将它们集中于高价值区域。这些提案通过低权重模仿损失和可选的软KL正则化器影响actor;重要的是,PPO梯度从未通过离线日志或纯合成轨迹回传。我们进一步从双近端视角分析该方法,并推导出一步性能下界。在八个统一的在线预算为100万环境步的MuJoCo连续控制任务中,PPO-DAP在早期学习效率(学习曲线前40个epoch的面积下积分,ALC@40)上持续改进,并在6/8个任务中匹配或超过最强的在线基线,在最终回报上表现优异,仅以适度的开销(1.18±0.04倍的墙钟时间和1.05±0.02倍的峰值GPU内存相对于PPO)为代价。

英文摘要:

Proximal Policy Optimization (PPO) is widely used in continuous control due to its robustness and stable training, yet it remains sample-inefficient in tasks with expensive interactions and high-dimensional action spaces. This paper proposes PPO-DAP (PPO with Diffusion Action Prior), a strictly on-policy framework that improves exploration quality and learning efficiency without modifying the PPO objective. PPO-DAP follows a two-stage protocol. Offline, we pretrain a conditional diffusion action prior on logged trajectories to cover the action distribution supported by the behavior policy. Online, PPO updates the actor-critic only using newly collected on-policy rollouts, while the diffusion prior is adapted around the on-policy state distribution via parameter-efficient tuning (Adapter/LoRA) over a small parameter subset. For each on-policy state, the prior generates multiple action proposals and concentrates them toward high-value regions using critic-based energy reweighting and in-denoising gradient guidance. These proposals affect the actor only through a low-weight imitation loss and an optional soft KL regularizer to the prior; importantly, PPO gradients are never backpropagated through offline logs or purely synthetic trajectories. We further analyze the method from a dual-proximal perspective and derive a one-step performance lower bound. Across eight MuJoCo continuous-control tasks under a unified online budget of 1.0M environment steps, PPO-DAP consistently improves early learning efficiency (area under the learning curve over the first 40 epochs, ALC@40) and matches or exceeds the strongest on-policy baselines in final return on 6/8 tasks, with modest overhead (1.18+/-0.04x wall-clock time and 1.05+/-0.02x peak GPU memory relative to PPO).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑