具有事后训练可达性验证的安全强化学习用于机器人导航
Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation
AI总结：
本文提出一种基于条件价值风险约束的强化学习框架，通过训练对高成本尾部结果敏感的策略，并在训练后利用神经网络可达性验证评估其安全边际，实验表明该方法在导航任务中实现了更高的安全验证率。
摘要：
移动机器人安全导航需要在高后果感知不确定性环境下保持可靠性的策略。然而，大多数现有安全强化学习（RL）方法通过平均累积成本评估安全性，此类指标可能掩盖危险的尾部风险行为。为此，我们提出了一种框架，通过在非策略TD3基础上进行条件价值风险（CVaR）约束优化训练风险敏感策略，并在训练后通过神经网络可达性验证评估其安全边际。在训练过程中，策略在累积成本上受CVaR约束优化，促进对高成本尾部结果的敏感性，而非仅平均行为。训练后，我们利用泰勒模型分析计算在有界观测不确定性下的可行动作集，得到一个量化评估状态中策略可行动作集保持在规定安全边际比例的安全率指标。关键发现是，受CVaR约束训练的策略在评估状态中对障碍物保持更大的安全边际。这使它们更易于进行形式可达性验证。在十个导航场景和六个基线方法的实验中，我们的方法实现了98.3%的成功率，是所有比较方法中最高的安全验证率，同时揭示了平均成本排名和基于可达性的安全排名可能产生分歧。这表明可达性验证捕捉到了仅被经验成本指标遗漏的风险。我们进一步在实际的Clearpath Jackal机器人上验证了我们的方法，展示了成功的仿真到现实迁移。
英文摘要：
Safe navigation for mobile robots demands policies that remain reliable under the high-consequence perception uncertainty of cluttered environments. Yet most existing safe reinforcement learning (RL) methods assess safety through average cumulative cost. Such metrics can mask dangerous tail-risk behaviors. To address this, we propose a framework that trains risk-sensitive policies through Conditional Value-at-Risk (CVaR) constrained optimization on an off-policy TD3 backbone and evaluates their safety margins post-training through neural network reachability verification. During training, the policy is optimized under CVaR constraints on cumulative costs, promoting sensitivity to high-cost tail outcomes rather than average behavior alone. After training, we compute action reachable sets under bounded observation uncertainty using Taylor Model analysis, yielding a safety rate metric that quantifies the proportion of evaluated states at which the policy's reachable action set remains within prescribed safety margins. A key finding is that policies trained with CVaR constraints maintain larger safety margins from obstacles across evaluated states. This makes them significantly more amenable to formal reachability verification. Experiments across ten navigation scenarios and six baselines show that our method achieves a 98.3\% success rate, the highest safety verification rate among all compared methods, while revealing that average cost rankings and reachability-based safety rankings can diverge. This indicates that reachability verification captures risks which are missed by empirical cost metrics alone. We further validate our approach on a physical Clearpath Jackal robot, demonstrating successful sim-to-real transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
