arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2105.02580cs.LG
收藏

时间感知Q网络:解决深度强化学习的时间不规则性

Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning

North Carolina State University(北卡罗来纳州立大学)

Yeo Jin Kim, Min Chi

摘要:

深度强化学习(DRL)在诱导有效的行动策略方面表现出出色的性能,这些策略可以最大化许多复杂任务的预期长期回报。大部分的日间行车灯的工作一直集中在离散的时间步长的事件序列,忽略了连续事件之间的不规则的时间间隔。鉴于在许多现实世界领域中,数据通常由具有不规则时间间隔的时间序列组成,因此考虑时间事件之间的时间间隔以捕获状态的潜在渐进模式是很重要的。在这项工作中,我们提出了一个通用的时间感知RL框架:时间感知Q网络(TQN),它考虑了深度RL框架内的物理时间间隔。TQN从两个方面处理时间不规则性:1)过去的流逝时间和预期的下一个观察时间,用于时间感知状态近似; 2)未来的动作时间窗口,用于时间感知奖励折扣。实验结果表明,TQN从两个方面都能捕捉到时间不规则序列的底层结构,在四种时间间隔不规则的背景下,TQN的性能明显优于DQN.更具体地说,我们的研究结果表明,在经典的RL任务,如CartPole和MountainCar和Atari基准与随机分段的时间间隔,时间感知折扣单独更重要,而在现实世界的任务,如核反应堆操作和脓毒症患者治疗与固有的时间间隔,时间感知状态和时间感知折扣都是至关重要的。此外,为了提高智能体的学习能力,我们探索了三种提升方法:双网络,决斗网络和优先经验重放,我们的结果表明,对于两个现实世界的任务,将所有三种提升方法与TQN相结合特别有效。

英文摘要:

Deep Reinforcement Learning (DRL) has shown outstanding performance on inducing effective action policies that maximize expected long-term return on many complex tasks. Much of DRL work has been focused on sequences of events with discrete time steps and ignores the irregular time intervals between consecutive events. Given that in many real-world domains, data often consists of temporal sequences with irregular time intervals, and it is important to consider the time intervals between temporal events to capture latent progressive patterns of states. In this work, we present a general Time-Aware RL framework: Time-aware Q-Networks (TQN), which takes into account physical time intervals within a deep RL framework. TQN deals with time irregularity from two aspects: 1) elapsed time in the past and an expected next observation time for time-aware state approximation, and 2) action time window for the future for time-aware discounting of rewards. Experimental results show that by capturing the underlying structures in the sequences with time irregularities from both aspects, TQNs significantly outperform DQN in four types of contexts with irregular time intervals. More specifically, our results show that in classic RL tasks such as CartPole and MountainCar and Atari benchmark with randomly segmented time intervals, time-aware discounting alone is more important while in the real-world tasks such as nuclear reactor operation and septic patient treatment with intrinsic time intervals, both time-aware state and time-aware discounting are crucial. Moreover, to improve the agent's learning capacity, we explored three boosting methods: Double networks, Dueling networks, and Prioritized Experience Replay, and our results show that for the two real-world tasks, combining all three boosting methods with TQN is especially effective.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑