基于后退时域安全评判器与可微动作护盾的安全强化学习
Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields
AI总结：
该研究提出结合后退时域安全评判器与可微动作护盾的安全强化学习方法，在四足运动模拟中提升了鲁棒性，减少了灾难性故障并保持了竞争力回报。
摘要：
尽管马尔可夫假设成立，机器人强化学习中的安全故障极少瞬时发生；它们通常会在多步后出现，原因是策略在干扰和域偏移下进入陌生状态。我们通过学习后退时域安全评判器来解决这一局限，该评判器可预测状态在接下来的后退时域内是否会保持非终止状态，从而实现对即将发生的运动故障的预判检测。该评判器通过从 rollout 得到的时域标签进行训练，采用的复合目标函数结合了时域分类、受控制 barrier 函数（CBF）启发的时间一致性以及平滑性正则化，并且我们聚合了一组评判器的集成，采用保守的条件风险价值（CVaR）估计来处理不确定性。基于学习到的评判器，我们推导了一个闭式、可微的 CBF 式安全过滤器，该过滤器会最小程度地修改策略动作以提升预测安全性，并将其集成到 PPO 中，同时加入预判损失以减少训练期间的策略-过滤器不匹配。在模拟四足运动任务中，我们的方法提升了在动作噪声、外部扰动和地形偏移下的鲁棒性，减少了灾难性故障，同时保持了具有竞争力的回报。
英文摘要：
Despite the Markov assumption, safety failures in robot reinforcement learning rarely occur instantaneously; they typically emerge over several steps as the policy enters unfamiliar states under disturbances and domain shift. We address this limitation by learning a receding-horizon safety critic that predicts whether a state will remain non-terminating over the next receding horizon, enabling anticipatory detection of impending locomotion failures. The critic is trained from rollout-derived horizon labels with a composite objective that combines horizon classification, control-barrier-function-inspired temporal consistency, and smoothness regularization, and we aggregate an ensemble of critics with a conservative Conditional Value at Risk estimate to account for uncertainty. Building on the learned critic, we derive a closed-form, differentiable CBF-style safety filter that minimally modifies the policy action to improve predicted safety, and integrate it into PPO with an anticipation loss to reduce policy-filter mismatch during training. In simulated quadruped locomotion, our method improves robustness under action noise, external perturbations, and terrain shifts, reducing catastrophic failures while maintaining competitive returns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
