arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.09035cs.AIcs.LG
收藏

基于优势的扩散模型用于基于模型的强化学习

Advantage-Guided Diffusion for Model-Based Reinforcement Learning

KTH Royal Institute of Technology(瑞典皇家理工学院);Digital Futures(数字未来)

Daniele Foffano, Arvid Eriksson, David Broman, Karl H. Johansson, Alexandre Proutiere

AI总结:

本文提出AGD-MBRL,通过利用智能体的优势估计引导反向扩散过程,提升长周期回报。采用SAG和EAG两种引导方法,改进了短周期视角下的扩散模型MBRL性能。

摘要:

基于模型的强化学习(MBRL)中,自回归世界模型因累积误差而受限,而扩散世界模型通过联合生成轨迹片段缓解此问题。然而,现有扩散引导要么仅依赖策略信息,要么基于奖励,当扩散时间较短时易出现短视问题。本文引入基于优势的扩散(AGD-MBRL),通过引导反向扩散过程,使采样集中在预期长期回报更高的轨迹上。开发了两种引导方法:(i) Sigmoid优势引导(SAG)和(ii) 指数优势引导(EAG)。证明通过SAG或EAG引导的扩散模型可实现加权采样,权重随状态-动作优势增加,暗示在标准假设下政策改进。此外,显示AGD-MBRL生成的轨迹比无引导的扩散模型具有更高的价值。AGD可无缝集成到PolyGRAD类架构中,引导状态组件的同时保留动作生成的策略条件化,无需改变扩散训练目标。在MuJoCo控制任务(HalfCheetah、Hopper、Walker2D和Reacher)中,AGD-MBRL在样本效率和最终回报上优于PolyGRAD(一种在线Diffuser式奖励引导)和模型无关基线(PPO/TRPO),在某些情况下提升幅度达2倍。这些结果表明,基于优势的引导是解决扩散模型MBRL中短周期短视问题的有效简单方法。

英文摘要:

Model-based reinforcement learning (MBRL) with autoregressive world models suffers from compounding errors, whereas diffusion world models mitigate this by generating trajectory segments jointly. However, existing diffusion guides are either policy-only, discarding value information, or reward-based, which becomes myopic when the diffusion horizon is short. We introduce Advantage-Guided Diffusion for MBRL (AGD-MBRL), which steers the reverse diffusion process using the agent's advantage estimates so that sampling concentrates on trajectories expected to yield higher long-term return beyond the generated window. We develop two guides: (i) Sigmoid Advantage Guidance (SAG) and (ii) Exponential Advantage Guidance (EAG). We prove that a diffusion model guided through SAG or EAG allows us to perform reweighted sampling of trajectories with weights increasing in state-action advantage-implying policy improvement under standard assumptions. Additionally, we show that the trajectories generated from AGD-MBRL follow an improved policy (that is, with higher value) compared to an unguided diffusion model. AGD integrates seamlessly with PolyGRAD-style architectures by guiding the state components while leaving action generation policy-conditioned, and requires no change to the diffusion training objective. On MuJoCo control tasks (HalfCheetah, Hopper, Walker2D and Reacher), AGD-MBRL improves sample efficiency and final return over PolyGRAD, an online Diffuser-style reward guide, and model-free baselines (PPO/TRPO), in some cases by a margin of 2x. These results show that advantage-aware guidance is a simple, effective remedy for short-horizon myopia in diffusion-model MBRL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑