arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.15537cs.LGcs.RO
收藏

约束下的外部罚函数策略优化与罚度量网络

Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints

Shanghai Jiao Tong University(上海交通大学)

Shiqing Gao, Jiaxin Ding, Luoyi Fu, Xinbing Wang, Chenghu Zhou

AI总结:

本文提出外部罚函数策略优化(EPO)方法,利用罚度量网络(PMN)自适应生成惩罚,在保证收敛的同时高效满足约束,实验证明其性能优于基线方法。

摘要:

在约束强化学习(CRL)中,智能体在满足约束条件的同时探索环境以学习最优策略。罚函数方法最近被研究为处理约束的有效方法,它通过对目标施加约束惩罚,将约束问题转化为无约束问题。然而,如何选择适当的惩罚以高效地平衡策略性能和约束满足度是一个挑战。在本文中,我们提出了一种具有理论保证的罚函数方法——外部罚函数策略优化(EPO),其自适应惩罚由罚度量网络(PMN)生成。PMN能对不同程度的约束违反做出适当响应,从而实现高效的约束满足和安全探索。我们从理论上证明了EPO能持续改善约束满足度,并具有收敛保证。我们提出了一种新的替代函数,并提供了最坏情况下的约束违反和近似误差。在实践中,我们提出了一种有效的平滑罚函数,可以很容易地用一阶优化器实现。我们进行了大量实验,结果表明,EPO在策略性能和约束满足度方面优于基线方法,且训练过程稳定,特别是在复杂任务上。

英文摘要:

In Constrained Reinforcement Learning (CRL), agents explore the environment to learn the optimal policy while satisfying constraints. The penalty function method has recently been studied as an effective approach for handling constraints, which imposes constraints penalties on the objective to transform the constrained problem into an unconstrained one. However, it is challenging to choose appropriate penalties that balance policy performance and constraint satisfaction efficiently. In this paper, we propose a theoretically guaranteed penalty function method, Exterior Penalty Policy Optimization (EPO), with adaptive penalties generated by a Penalty Metric Network (PMN). PMN responds appropriately to varying degrees of constraint violations, enabling efficient constraint satisfaction and safe exploration. We theoretically prove that EPO consistently improves constraint satisfaction with a convergence guarantee. We propose a new surrogate function and provide worst-case constraint violation and approximation error. In practice, we propose an effective smooth penalty function, which can be easily implemented with a first-order optimizer. Extensive experiments are conducted, showing that EPO outperforms the baselines in terms of policy performance and constraint satisfaction with a stable training process, particularly on complex tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑