arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.02620eess.SYcs.ROcs.SY
收藏

回归基座:通过具有可达-避碰安全滤波器的安全重置实现免人工学习

Back to Base: Towards Hands-Off Learning via Safe Resets with Reach-Avoid Safety Filters

University of California San Diego(加州大学圣地亚哥分校)

Azra Begzadić, Nikhil Uday Shinde, Sander Tonkens, Dylan Hirsch, Kaleb Ugalde, Michael C. Yip, Jorge Cortés, Sylvia Herbert

AI总结:

本文提出一种基于可达-避碰值函数的安全滤波器,通过最小修改名义控制器实现安全重置,从而支持现实机器人免人工强化学习,并在cartpole摆起任务上验证了方法。

摘要:

设计既能完成任务又能保证安全约束的控制器仍然是一个重大挑战。我们通常希望智能体在名义任务(例如环境探索)中表现良好,同时确保它能够避开不安全状态,并在特定时间之前返回期望目标。我们尤其受到现实世界中强化学习的安全、高效、免人工训练这一场景的激励。通过使机器人能够在没有人工干预的情况下安全且自主地重置到期望区域(例如充电站),我们可以提高效率并促进训练。安全滤波器,例如基于控制屏障函数的安全滤波器,将安全性与名义控制目标解耦,并严格保证安全。尽管它们取得了成功,但对于具有控制约束和系统不确定性的一般非线性系统,构造这些函数仍然是一个未解决的问题。本文引入了一种从可达-避碰问题相关的值函数中获得的安全滤波器。所提出的安全滤波器在避开不安全区域并引导系统返回期望目标集的同时,对名义控制器进行最小程度的修改。通过在允许安全重置的同时保持策略性能,我们实现了高效的免人工强化学习,并提高了现实世界机器人安全训练的可行性。我们使用修改版的soft actor-critic来演示我们的方法,在修改后的cartpole稳定问题上安全地训练一个摆起任务。

英文摘要:

Designing controllers that accomplish tasks while guaranteeing safety constraints remains a significant challenge. We often want an agent to perform well in a nominal task, such as environment exploration, while ensuring it can avoid unsafe states and return to a desired target by a specific time. In particular we are motivated by the setting of safe, efficient, hands-off training for reinforcement learning in the real world. By enabling a robot to safely and autonomously reset to a desired region (e.g., charging stations) without human intervention, we can enhance efficiency and facilitate training. Safety filters, such as those based on control barrier functions, decouple safety from nominal control objectives and rigorously guarantee safety. Despite their success, constructing these functions for general nonlinear systems with control constraints and system uncertainties remains an open problem. This paper introduces a safety filter obtained from the value function associated with the reach-avoid problem. The proposed safety filter minimally modifies the nominal controller while avoiding unsafe regions and guiding the system back to the desired target set. By preserving policy performance while allowing safe resetting, we enable efficient hands-off reinforcement learning and advance the feasibility of safe training for real world robots. We demonstrate our approach using a modified version of soft actor-critic to safely train a swing-up task on a modified cartpole stabilization problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑