arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.00546cs.LG
收藏

面向延迟强化学习的直接信念预测

Directly Forecasting Belief for Reinforcement Learning with Delays

University of Southampton(南安普顿大学);Northwestern University(西北大学);National Taiwan University(台湾大学);The Swiss AI Lab IDSIA/USI/SUPSI(瑞士人工智能实验室 IDSIA/USI/SUPSI)

Qingyuan Wu, Yuhui Wang, Simon Sinong Zhan, Yixuan Wang, Chung-Wei Lin, Chen Lv, Qi Zhu, Jürgen Schmidhuber, Chao Huang

AI总结:

针对延迟强化学习中递归状态预测的复合误差问题,提出DFBT直接由观测预测状态,理论上降低误差,并在D4RL与MuJoCo上提升预测、学习效率和整体性能。

摘要:

带延迟的强化学习(RL)具有挑战性,因为感官感知滞后于实际事件:RL智能体需要根据过去的观测来估计环境的真实状态。最先进(SOTA)方法通常采用递归的、逐步的状态预测,这会导致复合误差累积。为解决该问题,我们提出名为 Directly Forecasting Belief Transformer(DFBT)的新型信念估计方法,它直接从观测预测状态,而不是逐步增量地估计中间状态。我们从理论上证明,DFBT大幅降低了现有递归预测方法的复合误差,并带来更强的性能保证。在使用 D4RL 离线数据集的实验中,DFBT以显著的预测精度降低了复合误差。DFBT预测状态序列的能力还有助于多步自举,从而大幅提升学习效率。在 MuJoCo 基准上,我们基于 DFBT 的方法显著优于SOTA基线。代码可在 https://github.com/QingyuanWuNothing/DFBT 获取。

英文摘要:

Reinforcement learning (RL) with delays is challenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based on past observations. State-of-the-art (SOTA) methods typically employ recursive, step-by-step forecasting of states. This can cause the accumulation of compounding errors. To tackle this problem, our novel belief estimation method, named Directly Forecasting Belief Transformer (DFBT), directly forecasts states from observations without incrementally estimating intermediate states step-by-step. We theoretically demonstrate that DFBT greatly reduces compounding errors of existing recursively forecasting methods, yielding stronger performance guarantees. In experiments with D4RL offline datasets, DFBT reduces compounding errors with remarkable prediction accuracy. DFBT's capability to forecast state sequences also facilitates multi-step bootstrapping, thus greatly improving learning efficiency. On the MuJoCo benchmark, our DFBT-based method substantially outperforms SOTA baselines. Code is available at https://github.com/QingyuanWuNothing/DFBT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑