arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.11317cs.LGcs.AI
收藏

非平稳动态下用于快速在线适应的去偏离线表示学习

Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamics

National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Polixir Technologies(Polixir科技公司)

Xinyu Zhang, Wenjie Qiu, Yi-Chen Li, Lei Yuan, Chengxing Jia, Zongzhang Zhang, Yang Yu

AI总结:

针对非平稳环境下离线强化学习中的上下文误关联问题,提出去偏离线表示方法DORA,通过信息瓶颈原理分离环境动态与行为策略影响,在六个MuJoCo任务上取得更优性能。

摘要:

开发能够适应非平稳环境的策略对于现实世界的强化学习应用至关重要。然而,在离线设置下,仅利用有限的预收集轨迹来学习这种可适应的策略,面临着重大挑战。一个关键难点在于,有限的离线数据使得上下文编码器难以区分环境动态的变化与行为策略的偏移,常常导致上下文误关联。为解决这一问题,我们提出了一种新颖的方法,称为去偏离线表示用于快速在线适应(DORA)。DORA引入了信息瓶颈原理,该原理最大化动态编码与环境数据之间的互信息,同时最小化动态编码与行为策略动作之间的互信息。我们提出了DORA的一种实用实现,利用了信息瓶颈原理的可处理界限。我们在六个具有可变参数的MuJoCo基准任务上的实验评估表明,DORA不仅实现了更精确的动态编码,而且在性能上显著优于现有基线。

英文摘要:

Developing policies that can adjust to non-stationary environments is essential for real-world reinforcement learning applications. However, learning such adaptable policies in offline settings, with only a limited set of pre-collected trajectories, presents significant challenges. A key difficulty arises because the limited offline data makes it hard for the context encoder to differentiate between changes in the environment dynamics and shifts in the behavior policy, often leading to context misassociations. To address this issue, we introduce a novel approach called Debiased Offline Representation for fast online Adaptation (DORA). DORA incorporates an information bottleneck principle that maximizes mutual information between the dynamics encoding and the environmental data, while minimizing mutual information between the dynamics encoding and the actions of the behavior policy. We present a practical implementation of DORA, leveraging tractable bounds of the information bottleneck principle. Our experimental evaluation across six benchmark MuJoCo tasks with variable parameters demonstrates that DORA not only achieves a more precise dynamics encoding but also significantly outperforms existing baselines in terms of performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑