arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.05458cs.LG
收藏

警惕强化学习中的瞬时依赖

Beware of Instantaneous Dependence in Reinforcement Learning

Nanjing University(南京大学);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Zhengmao Zhu, Yuren Liu, Honglong Tian, Yang Yu, Kun Zhang

摘要:

环境模型在基于模型的强化学习(MBRL)中扮演着重要角色,其目标是基于过去预测未来状态。现有的工作通常忽略了状态的瞬时依赖性,即假设未来的状态变量是条件独立的给定的过去的状态。然而,瞬时依赖在许多RL环境中普遍存在。例如,在股票市场中,两支股票之间可能存在瞬时相关性,因为一只股票的波动可以迅速影响另一只股票,并且价格变化的分辨率低于影响的分辨率。在本文中,我们证明了,除了少数例外,忽略瞬时依赖可能会导致次优的策略学习MBRL。为了解决次优问题,我们提出了一个简单的即插即用的方法,使现有的MBRL算法考虑瞬时依赖。通过对两个基准测试的实验,我们(1)通过可视化证实了瞬时依赖的存在;(2)验证了我们的理论发现,即忽略瞬时依赖会导致次优策略;(3)验证了我们的方法有效地实现了具有瞬时依赖的强化学习,并提高了策略性能。

英文摘要:

Playing an important role in Model-Based Reinforcement Learning (MBRL), environment models aim to predict future states based on the past. Existing works usually ignore instantaneous dependence in the state, that is, assuming that the future state variables are conditionally independent given the past states. However, instantaneous dependence is prevalent in many RL environments. For instance, in the stock market, instantaneous dependence can exist between two stocks because the fluctuation of one stock can quickly affect the other and the resolution of price change is lower than that of the effect. In this paper, we prove that with few exceptions, ignoring instantaneous dependence can result in suboptimal policy learning in MBRL. To address the suboptimality problem, we propose a simple plug-and-play method to enable existing MBRL algorithms to take instantaneous dependence into account. Through experiments on two benchmarks, we (1) confirm the existence of instantaneous dependence with visualization; (2) validate our theoretical findings that ignoring instantaneous dependence leads to suboptimal policy; (3) verify that our method effectively enables reinforcement learning with instantaneous dependence and improves policy performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑