不确定离散系统的连续深度Q学习镇定
Continuous Deep Q-Learning with Simulator for Stabilization of Uncertain Discrete-Time Systems
摘要：
强化学习（RL）在实际系统的镇定问题中的应用受到限制，因为智能体需要许多经验来学习最优策略，并且可能在其探索过程中确定危险行为。如果我们知道真实系统的数学模型，则仿真器是有用的，因为它使用具有给定系统参数向量的数学模型来预测真实系统的行为。我们可以比与真实系统交互更有效地收集许多经验。然而，很难准确地识别系统参数向量。如果我们有一个识别错误，由模拟器获得的经验可能会降低学习策略的性能。因此，我们提出了一个实用的RL算法，包括两个阶段。在第一阶段，我们选择多个系统参数向量。然后，我们为每个系统参数向量建立了一个数学模型，称为虚拟系统。我们使用连续深度Q-学习算法获得多个虚拟系统的最优Q函数。在第二阶段，我们用一个线性近似函数来表示真实系统的Q函数，该线性近似函数的基函数是在第一阶段学习的最优Q函数。智能体通过与真实系统的在线交互来学习Q函数。通过数值模拟，我们证明了我们所提出的方法的实用性。
英文摘要：
Applications of reinforcement learning (RL) to stabilization problems of real systems are restricted since an agent needs many experiences to learn an optimal policy and may determine dangerous actions during its exploration. If we know a mathematical model of a real system, a simulator is useful because it predicates behaviors of the real system using the mathematical model with a given system parameter vector. We can collect many experiences more efficiently than interactions with the real system. However, it is difficult to identify the system parameter vector accurately. If we have an identification error, experiences obtained by the simulator may degrade the performance of the learned policy. Thus, we propose a practical RL algorithm that consists of two stages. At the first stage, we choose multiple system parameter vectors. Then, we have a mathematical model for each system parameter vector, which is called a virtual system. We obtain optimal Q-functions for multiple virtual systems using the continuous deep Q-learning algorithm. At the second stage, we represent a Q-function for the real system by a linear approximated function whose basis functions are optimal Q-functions learned at the first stage. The agent learns the Q-function through interactions with the real system online. By numerical simulations, we show the usefulness of our proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
