arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.11986eess.SYcs.SY
收藏

基于线性矩阵不等式的数据高效二次Q学习

Data-Efficient Quadratic Q-Learning Using LMIs

Eindhoven University of Technology(埃因霍温理工大学)

J. S. van Hulst, W. P. M. H. Heemels, D. J. Antunes

AI总结:

针对强化学习数据需求大的问题,提出LMI-QL及LMI-QLi两种数据高效离线策略Q学习方法,结合凸优化与直接Q学习提升速度,经数值案例验证其性能优于现有同类方法。

摘要:

强化学习(RL)已取得显著研究与应用成果,但通常需要大量训练数据。本文提出两种数据高效的离线策略强化学习方法,采用参数化Q学习。在这些方法中,Q函数对参数呈线性,对状态和控制相对于基础策略的偏差的选定基函数呈二次。最小化惩罚Bellman误差的ℓ₁范数的代价,提出两种方法:线性矩阵不等式Q学习(LMI-QL)及其迭代变体(LMI-QLi),通过凸优化求解所得的情节优化问题。LMI-QL依赖凸松弛,得到含线性矩阵不等式(LMIs)的半定规划(SDP)问题;LMI-QLi需求解SDP问题的序列迭代。两种方法均将凸优化与直接Q函数学习相结合,显著提升学习速度。数值案例研究表明,它们优于现有参数化Q学习方法。

英文摘要:

Reinforcement learning (RL) has seen significant research and application results but often requires large amounts of training data. This paper proposes two data-efficient off-policy RL methods that use parametrized Q-learning. In these methods, the Q-function is chosen to be linear in the parameters and quadratic in selected basis functions in the state and control deviations from a base policy. A cost penalizing the $\ell_1$-norm of Bellman errors is minimized. We propose two methods: Linear Matrix Inequality Q-Learning (LMI-QL) and its iterative variant (LMI-QLi), which solve the resulting episodic optimization problem through convex optimization. LMI-QL relies on a convex relaxation that yields a semidefinite programming (SDP) problem with linear matrix inequalities (LMIs). LMI-QLi entails solving sequential iterations of an SDP problem. Both methods combine convex optimization with direct Q-function learning, significantly improving learning speed. A numerical case study demonstrates their advantages over existing parametrized Q-learning methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑