HypeRL: 基于超网络的强化学习用于参数化动力系统控制
HypeRL: Hypernetwork-Based Reinforcement Learning for Control of Parametrized Dynamical Systems
AI总结：
HypeRL通过超网络强化学习方法，解决参数化动力系统控制问题，实现高效泛化和优化。
摘要：
在本文中，我们提出了一种新的通用强化学习策略，用于参数化动力系统的最优控制。此类问题经常出现在应用科学和工程中，当控制变量或状态变量分布在高维空间或依赖于变化参数时，其复杂性显著增加。传统数值方法依赖于迭代最小化算法（例如利用伴随问题的解）或动态规划（也涉及哈密顿-雅可比-贝尔曼方程的解）的解决方法，虽然可靠，但往往在计算上不可行。在本文中，我们提出了HypeRL，一种深度强化学习（DRL）框架，以克服传统方法的限制。HypeRL旨在直接近似最优控制策略。具体而言，我们采用actor-critic DRL方法来学习一个能够跨参数变化范围泛化的最优反馈控制策略。为了有效学习不同参数实例的最优控制律，将参数信息编码到DRL策略和价值函数神经网络（NNs）中是至关重要的。HypeRL使用两个额外的NNs，称为超网络，来学习价值函数和策略NNs的权重和偏差。通过这种方式，HypeRL有效地将参数信息嵌入到价值函数和策略中。我们验证了所提出的方法在两个参数化控制问题上的有效性，即（I）一个带有域内控制的1D参数化库拉托夫斯基-西瓦斯基方程，以及（ii）粒子动力学在参数化2D环流中的导航问题。我们展示了物理和任务依赖信息的知识以及通过超网络编码这种信息是学习参数依赖控制策略的关键要素。
英文摘要：
In this work, we devise a new, general-purpose reinforcement learning strategy for the optimal control of parametric dynamical systems. Such problems frequently arise in applied sciences and engineering and entail a significant complexity when control and/or state variables are distributed in high-dimensional space or depend on varying parameters. Traditional numerical methods, relying on either iterative minimization algorithms -- exploiting, e.g., the solution of the adjoint problem -- or dynamic programming -- also involving the solution of the Hamilton-Jacobi-Bellman (HJB) equation -- while reliable, often become computationally infeasible. In this paper, we propose HypeRL a deep reinforcement learning (DRL) framework to overcome the limitations shown by traditional methods. HypeRL aims at approximating the optimal control policy directly. Specifically, we employ an actor-critic DRL approach to learn an optimal feedback control strategy that can generalize across the range of variation of the parameters. To effectively learn such optimal control laws for different instances of the parameters, encoding the parameter information into the DRL policy and value function neural networks (NNs) is essential. HypeRL uses two additional NNs, called hypernetworks, to learn the weights and biases of the value function and the policy NNs. In this way, HypeRL effectively embeds the parametric information into the value function and policy. We validate the proposed approach on two parametric control problems, namely (I) a 1D parametric Kuramoto-Sivashinsky equation with in-domain control, and (ii) a navigation problem of particle dynamics in a parametric 2D gyre flow. We show that the knowledge of physical and task-dependent information and the encoding of this information via a hypernetwork, are essential ingredients for learning parameter-dependent control policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
