arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.16475eess.SYcs.SY
收藏

通过强化学习学习的汉密尔顿-雅可比可达性认证

Certifying Hamilton-Jacobi Reachability Learned via Reinforcement Learning

Delft University of Technology(代尔夫特理工大学)

Prashant Solanki, Isabelle El-Hajj, Jasper J. van Beers, Erik-Jan van Kampen, Coen C. de Visser

AI总结:

通过强化学习学习的汉密尔顿-雅可比可达性认证方法,利用加法偏移恒等式和两种界方法,实现对价值函数的正式认证。

摘要:

我们提出一个框架,用于通过强化学习(RL)学习的汉密尔顿-雅可比(HJ)可达性认证。基于一种折扣初始时间旅行成本公式,该公式使小步RL价值迭代与正向汉密尔顿-雅可比(HJ)方程具有阻尼效应的证明等价。我们将认证学习误差转换为校准的内/外严格反向可达管的内/外包围。核心设备是加法偏移恒等式:如果W_λ解决了折扣旅行成本汉密尔顿-雅可比-贝尔曼(HJB)方程,则W_ε:=W_λ + ε解决了相同的PDE,具有常数偏移λε。这意味着统一的价值误差恰好等于一个常数HJB偏移。我们通过两种途径建立这种统一的价值误差:(A)贝尔曼算子残差界,和(B)HJB PDE松弛界。我们的框架保持HJ级别的安全语义,并且与深度RL兼容。我们通过在双积分系统上正式认证,通过可满足性模理论(SMT)证明通过强化学习学习的价值函数在感兴趣的紧致区域上诱导出可证明正确的内/外反向可达集包围。

英文摘要:

We present a framework to \emph{certify} Hamilton--Jacobi (HJ) reachability learned by reinforcement learning (RL). Building on a discounted initial time \emph{travel-cost} formulation that makes small-step RL value iteration provably equivalent to a forward Hamilton--Jacobi (HJ) equation with damping, we convert certified learning errors into calibrated inner/outer enclosures of strict backward reachable tube. The core device is an additive-offset identity: if $W_λ$ solves the discounted travel-cost Hamilton--Jacobi--Bellman (HJB) equation, then $W_\varepsilon:=W_λ+ \varepsilon$ solves the same PDE with a constant offset $λ\varepsilon$. This means that a uniform value error is \emph{exactly} equal to a constant HJB offset. We establish this uniform value error via two routes: (A) a Bellman operator-residual bound, and (B) a HJB PDE-slack bound. Our framework preserves HJ-level safety semantics and is compatible with deep RL. We demonstrate the approach on a double-integrator system by formally certifying, via satisfiability modulo theories (SMT), a value function learned through reinforcement learning to induce provably correct inner and outer backward-reachable set enclosures over a compact region of interest.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑