基于神经网络的庞特里亚金最优控制
Pontryagin Optimal Control via Neural Networks
AI总结:
本文提出 NN-PMP-Gradient 框架,将神经网络与庞特里亚金极大值原理结合,以样本高效方式求解动态未知复杂系统的最优控制问题,并在多个控制任务中优于强化学习方法。
摘要:
解决现实世界的最优控制问题是一项具有挑战性的任务,因为复杂、高维的系统动态通常对决策者而言是未知的。因此,很难通过数值方法找到最优控制动作。为了应对这类建模与计算挑战,在本文中,我们将神经网络与庞特里亚金极大值原理(PMP)相结合,提出了一个样本高效的框架 NN-PMP-Gradient。由此得到的控制器可应用于动态未知且复杂的系统。通过采用迭代方法,所提出的框架不仅利用了由神经网络参数化的精确代理模型,还通过 PMP 条件高效地恢复最优性条件以及最优动作序列。在线性二次调节器、并网有损电池的能量套利、单摆控制以及两个 MuJoCo 运动任务上的数值仿真表明,我们提出的 NN-PMP-Gradient 是一种通用且多功能的计算工具,可用于寻找最优解。与广泛应用的免模型和基于模型的强化学习(RL)算法相比,我们的 NN-PMP-Gradient 在控制目标方面实现了更高的样本效率和性能。
英文摘要:
Solving real-world optimal control problems are challenging tasks, as the complex, high-dimensional system dynamics are usually unrevealed to the decision maker. It is thus hard to find the optimal control actions numerically. To deal with such modeling and computation challenges, in this paper, we integrate Neural Networks with the Pontryagin's Maximum Principle (PMP), and propose a sample efficient framework NN-PMP-Gradient. The resulting controller can be implemented for systems with unknown and complex dynamics. By taking an iterative approach, the proposed framework not only utilizes the accurate surrogate models parameterized by neural networks, it also efficiently recovers the optimality conditions along with the optimal action sequences via PMP conditions. Numerical simulations on Linear Quadratic Regulator, energy arbitrage of grid-connected lossy battery, control of single pendulum, and two MuJoCo locomotion tasks demonstrate our proposed NN-PMP-Gradient is a general and versatile computation tool for finding optimal solutions. And compared with the widely applied model-free and model-based reinforcement learning (RL) algorithms, our NN-PMP-Gradient achieves higher sample-efficiency and performance in terms of control objectives.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。