arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.12932cs.LG
收藏

通过接受-拒绝方法和增广MDP实现高效动作约束强化学习

Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs

National Yang Ming Chiao Tung University(国立阳明交通大学);National Taiwan University(国立台湾大学)

Wei Hung, Shao-Hua Sun, Ping-Chun Hsieh

AI总结:

针对动作约束RL中QP计算昂贵和生成模型复杂的问题,提出结合接受-拒绝采样与含自环、拒绝惩罚的增广双目标MDP的高效框架。

摘要:

动作约束强化学习(ACRL)是一种通用框架,用于学习实现零动作约束违反的控制策略,这是各类安全关键和资源受限应用所必需的。现有ACRL方法通常能够获得良好的约束满足效果,但代价要么是二次规划(QP)带来的高计算负担,要么是由于使用复杂生成模型而增加的架构复杂性。本文提出一种通用且计算高效的框架,可通过两项修改使标准无约束RL方法适配ACRL:(i)为强制执行动作约束,我们利用经典的接受-拒绝方法,将无约束策略视为提议分布,并推导出具有可行动作的修改策略。(ii)为提高提议分布的接受率,我们构建了一个增广的双目标Markov决策过程(MDP),其中包含额外的自环状态转移以及针对被拒绝动作的惩罚信号。该增广MDP激励所学策略保持接近可行动作集。通过在机器人控制和资源分配领域的大量实验,我们证明所提框架相比最先进的ACRL方法,同时具有更快的训练进展、更好的约束满足以及更低的动作推理时间。我们已公开源代码,以鼓励沿此方向的进一步研究。

英文摘要:

Action-constrained reinforcement learning (ACRL) is a generic framework for learning control policies with zero action constraint violation, which is required by various safety-critical and resource-constrained applications. The existing ACRL methods can typically achieve favorable constraint satisfaction but at the cost of either high computational burden incurred by the quadratic programs (QP) or increased architectural complexity due to the use of sophisticated generative models. In this paper, we propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL through two modifications: (i) To enforce the action constraints, we leverage the classic acceptance-rejection method, where we treat the unconstrained policy as the proposal distribution and derive a modified policy with feasible actions. (ii) To improve the acceptance rate of the proposal distribution, we construct an augmented two-objective Markov decision process (MDP), which include additional self-loop state transitions and a penalty signal for the rejected actions. This augmented MDP incentives the learned policy to stay close to the feasible action sets. Through extensive experiments in both robot control and resource allocation domains, we demonstrate that the proposed framework enjoys faster training progress, better constraint satisfaction, and a lower action inference time simultaneously than the state-of-the-art ACRL methods. We have made the source code publicly available to encourage further research in this direction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑