数据驱动哈密顿量用于神经最优控制
Data-Driven Hamiltonian for Neural Optimal Control
AI总结:
该研究提出数据驱动哈密顿量(DDH)方法,结合哈密顿-雅可比框架,在LQR和月球着陆器基准测试中,以远少于现有RL方法的样本量实现更优性能,为数据有限场景的最优控制提供了 promising 框架。
摘要:
强化学习(RL)已成为最优控制领域中一种突出的数据驱动方法,但通常需要大量数据,且在遇到分布外状态-动作对时能提供的保证有限。本文提出一种基于最优控制理论中成熟的哈密顿-雅可比框架的替代方法。我们不要求显式的系统动力学解析模型来评估哈密顿量,而是用数据驱动哈密顿量(DDH)替代该模型,并将其推广到广泛的最优控制问题类别中。与标准RL方法不同,DDH提供了一种原则性机制,仅需最小的先验系统信息(如利普希茨常数),即可量化并考虑数据未覆盖区域的不确定性,从而推导具有性能保证的控制器。若有额外的先验系统知识,可直接纳入以提升性能。对于高维系统,我们进一步引入价值函数的物理信息神经近似,采用通过DDH生成最坏情况虚构滚动的新流程进行训练。在LQR问题上,DDH在我们测试的每个数据集上都达到了接近最优的代价,包括离线RL方法仍远未达到最优的随机数据,且所需样本远少于在线RL。在非线性月球着陆器基准测试中,DDH与轨迹优化参考的最优性差距比CQL和FQL小3倍以上,且以约37倍更少的样本达到与SAC相当的性能。这些结果表明,DDH是一种有前景的数据驱动最优控制框架,尤其适用于可用数据数量或质量有限的场景。
英文摘要:
Reinforcement learning (RL) has emerged as a prominent data-driven approach to optimal control, but suffers from sample inefficiency and sensitivity to the data distribution. In this paper, we propose an alternative approach grounded in the well-established Hamilton-Jacobi framework in optimal control theory. Rather than requiring an explicit analytical model of the system dynamics to evaluate the Hamiltonian, we replace this with a Data-Driven Hamiltonian (DDH) that can be applied to a broad class of optimal control problems. In contrast to standard RL approaches, the DDH provides a principled mechanism for quantifying and accounting for uncertainty in regions not covered by the data, requiring only minimal prior system information such as a Lipschitz constant. Additional prior system knowledge, when available, can be readily incorporated to improve performance. For higher-dimensional systems, we further introduce a physics-informed neural approximation of the value function, trained using a novel procedure that generates worst-case fictitious rollouts from the DDH. On an LQR problem, the DDH reaches near-optimal cost on every dataset we test, including random data on which offline RL methods remain far from optimal, and needs far fewer samples than online RL. On a nonlinear lunar lander benchmark, the DDH achieves an optimality gap relative to a trajectory-optimization reference that is less than one-third of those achieved by CQL and FQL, while matching SAC's performance with approximately $37\times$ fewer samples.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。