arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.02182cs.LGcs.AI
收藏

非平稳上下文驱动环境中的在线强化学习

Online Reinforcement Learning in Non-Stationary Context-Driven Environments

MIT(麻省理工学院);CS Brown University(布朗大学计算机科学系);Microsoft Research(微软研究院)

Pouya Hamadanian, Arash Nasr-Esfahany, Malte Schwarzkopf, Siddartha Sen, Mohammad Alizadeh

AI总结:

针对非平稳环境中在线强化学习的灾难性遗忘问题,提出局部约束策略优化(LCPO)方法,通过锚定旧经验策略输出并在当前上下文分布外采样进行局部约束,显著提升了性能并与离线先知智能体表现相当。

摘要:

我们研究非平稳环境中的在线强化学习(RL),其中时变的外生上下文过程会影响环境动态。由于“灾难性遗忘”(CF),此类环境中的在线RL极具挑战性。智能体在基于新经验训练时,往往会遗忘先前的知识。缓解该问题的现有方法假设存在任务标签(实践中通常不可用),采用脆弱的正则化启发式策略,或使用存在不稳定性和性能较差的离策略方法。我们提出了局部约束策略优化(LCPO),这是一种在线RL方法,它通过在优化当前经验回报的同时锚定旧经验上的策略输出来对抗CF。为执行此锚定,LCPO利用位于当前上下文分布之外的经验样本对策略优化进行局部约束。我们在Mujoco、经典控制和计算机系统环境中,使用各种合成和真实的上下文轨迹对LCPO进行了评估,发现它在非平稳环境中优于多种基线方法,同时在所有上下文轨迹上取得了与离线训练的“先知”智能体相当的结果。LCPO的源代码可在https://github.com/pouyahmdn/LCPO获取。

英文摘要:

We study online reinforcement learning (RL) in non-stationary environments, where a time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to "catastrophic forgetting" (CF). The agent tends to forget prior knowledge as it trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice), employ brittle regularization heuristics, or use off-policy methods that suffer from instability and poor performance. We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems environments with a variety of synthetic and real context traces, and find that it outperforms a variety of baselines in the non-stationary setting, while achieving results on-par with a "prescient" agent trained offline across all context traces. LCPO's source code is available at https://github.com/pouyahmdn/LCPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑