arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.03489eess.SYcs.SY
收藏

基于深度强化学习的概率布尔控制网络最优无限时域控制

Deep Reinforcement Learning Based Optimal Infinite-Horizon Control of Probabilistic Boolean Control Networks

Jingjie Ni, Fangfei Li, Zheng-Guang Wu

AI总结:

针对概率布尔控制网络的最优无限时域控制问题,提出一种基于深度强化学习的无模型方法,将问题转化为马尔可夫决策过程,并分别利用Q-learning和双深度Q网络设计适用于小规模和大规模网络的最优状态反馈控制器。

摘要:

本文提出了一种基于深度强化学习的方法,用于求解概率布尔控制网络(PBCNs)最优无限时域控制的最优策略。与现有文献相比,所提方法是无模型的,即无需已知系统模型和初始状态。同时,该方法适用于大规模PBCNs。首先,我们建立了深度强化学习与最优无限时域控制之间的联系,并将该问题构建为马尔可夫决策过程的框架。然后,根据动作值的内存是否超过计算机的RAM,将PBCNs定义为大规模或小规模。基于新引入的定义,Q-learning(QL)和双深度Q网络(DDQN)分别被应用于小规模和大规模PBCNs的最优无限时域控制。同时,设计了最优状态反馈控制器。最后,给出了两个例子,分别是具有3个节点的小规模PBCN和具有28个节点的大规模PBCN。为了验证QL和DDQN的收敛性,将这两种算法获得的最优控制策略和最优动作值与基于名为策略迭代的模型方法获得的结果进行了比较。同时,在小规模PBCN中比较了QL和DDQN的性能。

英文摘要:

In this paper, a deep reinforcement learning based method is proposed to obtain optimal policies for optimal infinite-horizon control of probabilistic Boolean control networks (PBCNs). Compared with the existing literatures, the proposed method is model-free, namely, the system model and the initial states needn't to be known. Meanwhile, it is suitable for large-scale PBCNs. First, we establish the connection between deep reinforcement learning and optimal infinite-horizon control, and structure the problem into the framework of the Markov decision process. Then, PBCNs are defined as large-scale or small-scale, depending on whether the memory of the action-values exceeds the RAM of the computer. Based on the newly introduced definition, Q-learning (QL) and double deep Q-network (DDQN) are applied to the optimal infinite-horizon control of small-scale and large-scale PBCNs, respectively. Meanwhile, the optimal state feedback controllers are designed. Finally, two examples are presented, which are a small-scale PBCN with 3 nodes, and a large-scale one with 28 nodes. To verify the convergence of QL and DDQN, the optimal control policy and the optimal action-values, which are obtained from both the algorithms, are compared with the ones based on a model-based method named policy iteration. Meanwhile, the performance of QL is compared with DDQN in the small-scale PBCN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑