Software agents built on large language models act by issuing tool calls, some of which cannot be undone. To control theory the model is an opaque feedback controller, replaced every few months, whose closed loop cannot be analyzed directly. We propose a runtime filter, or shield, between any such controller and its environment that certifies a bound on the probability of reaching an unsafe condition during a bounded task. Given a predicate abstraction and candidate successor sets, everything else comes from logs: the loop becomes a finite controlled Markov chain, the estimated kernel is surrounded by a distribution-free confidence region, and robust reach-avoid value iteration runs over it. The certificate is probably approximately correct and uniform over controllers respecting the filter, so one model may replace another without recomputation. In all tested datasets the true risk stayed within certificate and budget; in a file-system sandbox the filter stays sound against a hazard the logs barely covered, where a plug-in baseline certifies safety and fails; and it holds seven controllers, three of them open-weights models, to zero unsafe outcomes in 1050 episodes.
Tsinghua University（清华大学）；Harvard University（哈佛大学）；University of Science and Technology Beijing（北京科技大学）
Safe reinforcement learning (RL) that solves constraint-satisfactory policies provides a promising way to the broader safety-critical applications of RL in real-world problems such as robotics. Among all safe RL approaches, model-based methods reduce training time violations further due to their high sample efficiency. However, lacking safety robustness against the model uncertainties remains an issue in safe model-based RL, especially in training time safety. In this paper, we propose a distributional reachability certificate (DRC) and its Bellman equation to address model uncertainties and c…
Runtime Monitoring of Supermartingale Certificates
University of Birmingham（伯明翰大学）；Institute of Science and Technology Austria（奥地利科学技术研究所）；University of Oxford（牛津大学）；Leiden University（莱顿大学）；Nanyang Technological University（南洋理工大学）
We investigate a linear quadratic Gaussian (LQG) tracking problem with safety and reachability constraints in the presence of an adversary who mounts an FDI attack on an unknown set of sensors. For each possible set of compromised sensors, we maintain a state estimator disregarding the sensors in that set, and calculate the optimal LQG control input at each time based on this estimate. We propose a control policy which constrains the control input to lie within a fixed distance of the optimal control input corresponding to each state estimate. The control input is obtained at each time step by…
Anytime-Valid Runtime Monitoring of Learned Barrier Certificates
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学前沿交叉科学学院）；University of Chinese Academy of Sciences（中国科学院大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）
This paper is concerned with a compositional approach for the construction of control barrier certificates for large-scale interconnected stochastic systems while synthesizing hybrid controllers against high-level logic properties. Our proposed methodology involves decomposition of interconnected systems into smaller subsystems and leverages the notion of control sub-barrier certificates of subsystems, enabling one to construct control barrier certificates of interconnected systems by employing some max-type small-gain conditions. The main goal is to synthesize hybrid controllers enforcing com…
Safe Reach Set Computation via Neural Barrier Certificates
University of Oxford（牛津大学）；Newcastle University（纽卡斯尔大学）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；Università di Roma “La Sapienza”（罗马第一大学）
This paper works towards unifying two popular approaches in the safety control community: Hamilton-Jacobi (HJ) reachability and Control Barrier Functions (CBFs). HJ Reachability has methods for direct construction of value functions that provide safety guarantees and safe controllers, however the online implementation can be overly conservative and/or rely on chattering bang-bang control. The CBF community has methods for safe-guarding controllers in the form of point-wise optimization using quadratic programs (CBF-QP), where the CBF-based safety certificate is used as a constraint. However, f…
Quadratic Characterizations for Reachability Analysis of Neural Networks
Kevin T. Crofton Department of Aerospace and Ocean Engineering, Virginia Tech（凯文·T·克罗夫顿航空航天与海洋工程系，弗吉尼亚理工学院）；Federation ENAC ISAE-SUPAERO ONERA, Universite de Toulouse（ENAC ISAE-SUPAERO ONERA联盟，图卢兹大学）
School of Mechanical Engineering, University of Michigan（密歇根大学机械工程学院）；Robotics Institute, University of Michigan（密歇根大学机器人研究所）；Stanford University（斯坦福大学）
Reinforcement Learning (RL) algorithms have achieved remarkable performance in decision making and control tasks due to their ability to reason about long-term, cumulative reward using trial and error. However, during RL training, applying this trial-and-error approach to real-world robots operating in safety critical environment may lead to collisions. To address this challenge, this paper proposes a Reachability-based Trajectory Safeguard (RTS), which leverages reachability analysis to ensure safety during training and operation. Given a known (but uncertain) model of a robot, RTS precompute…
Recent advances in machine learning technologies and sensing have paved the way for the belief that safe, accessible, and convenient autonomous vehicles may be realized in the near future. Despite tremendous advances within this context, fundamental challenges around safety and reliability are limiting their arrival and comprehensive adoption. Autonomous vehicles are often tasked with operating in dynamic and uncertain environments. As a result, they often make use of highly complex components, such as machine learning approaches, to handle the nuances of sensing, actuation, and control. While…
Myopically Verifiable Probabilistic Certificates for Safe Control and Learning
Department of Electrical and Computer Engineering, Carnegie Mellon University（电气与计算机工程系，卡内基梅隆大学）；Information Systems Technology and Design pillar, Singapore University of Technology and Design（信息系统技术与设计支柱，新加坡科技设计大学）；Department of Computer Science and Engineering, University of California San Diego（计算机科学与工程系，加州大学圣地亚哥分校）
MIT（麻省理工学院）；Toyota Research Institute（丰田研究所）；Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）
Widespread adoption of autonomous cars will require greater confidence in their safety than is currently possible. Certified control is a new safety architecture whose goal is two-fold: to achieve a very high level of safety, and to provide a framework for justifiable confidence in that safety. The key idea is a runtime monitor that acts, along with sensor hardware and low-level control and actuators, as a small trusted base, ensuring the safety of the system as a whole. Unfortunately, in current systems complex perception makes the verification even of a runtime monitor challenging. Unlike tr…
RTD-RAX: Fast, Safe Trajectory Planning for Systems under Unknown Disturbances
Georgia Institute of Technology（佐治亚理工学院）；School of Electrical and Computer Engineering（电子与计算机工程学院）；School of Mechanical Engineering（机械工程学院）；School of Civil and Environmental Engineering（土木与环境工程学院）
Context-Triggered Robust MPC for Temporal Logic Specifications
Max Planck Institute for Software Systems (MPI-SWS)（马克斯·普朗克软件系统研究所）；Institute of Science and Technology Austria (ISTA)（奥地利科学技术研究院）；University of Birmingham（伯明翰大学）
