This paper proposes a framework based on a causal model of safety upon which effective safety assurance cases for ML-based applications can be built. In doing so, we build upon established principles of safety engineering as well as previous work on structuring assurance arguments for ML. The paper defines four categories of safety case evidence and a structured analysis approach within which these evidences can be effectively combined. Where appropriate, abstract formalisations of these contributions are used to illustrate the causalities they evaluate, their contributions to the safety argument and desirable properties of the evidences. Based on the proposed framework, progress in this area is re-evaluated and a set of future research directions proposed in order for tangible progress in this field to be made.
Integrated safety and security assurance for complex systems is difficult for many technical and socio-technical reasons such as mismatched processes, inadequate information, differing use of language and philosophies, etc.. Many co-assurance techniques rely on disregarding some of these challenges in order to present a unified methodology. Even with this simplification, no methodology has been widely adopted primarily because this approach is unrealistic when met with the complexity of real-world system development. This paper presents an alternate approach by providing a Safety-Security Assu…
University of Liverpool（利物浦大学）；Heriot-Watt University（赫瑞-瓦特大学）；Defence Science and Technology Laboratory（国防科学技术实验室）；Imperial College London（帝国理工学院）
The increasing use of Machine Learning (ML) components embedded in autonomous systems -- so-called Learning-Enabled Systems (LESs) -- has resulted in the pressing need to assure their functional safety. As for traditional functional safety, the emerging consensus within both, industry and academia, is to use assurance cases for this purpose. Typically assurance cases support claims of reliability in support of safety, and can be viewed as a structured way of organising arguments and evidence generated from safety analysis and reliability modelling activities. While such assurance activities ar…
In this work, we outline a cross-domain assurance process for safety-relevant software in embedded systems. This process aims to be applied in various different application domains and in conjunction with any development methodology. With this approach we plan to reduce the growing effort for safety assessment in embedded systems by reusing safety analysis techniques and tools for the product development in different domains.
The usage of complex Microcontroller Units (MCUs) in avionic systems constitutes a challenge in assuring their safety. They are not developed according to the development requirements accepted by the aerospace industry. These Commercial off-the-shelf (COTS) hardware components usually target other domains like the telecommunication branch. In the last years MCUs developed in compliance to the ISO 26262 have been released on the market for safety-related automotive applications. The avionic assurance process could profit from these safety MCUs. In this paper we present evaluation results based …
The specification, design, and assurance of safety encompasses various concepts and best practices, subject of reuse in form of patterns. This work summarizes applied research on such concepts and practices with a focus on the last two decades and on the state-of-the-art of patterns in safety-critical system design and assurance argumentation. We investigate several aspects of such patterns, for example, where and when they are applied, their characteristics and purposes, and how they are related. For each aspect, we provide an overview of relevant studies and synthesize a taxonomy of first pr…
Self-driving Autonomous Vehicles (SAVs) are gaining more interest each passing day by the industry as well as the general public. Tech and automobile companies are investing huge amounts of capital in research and development of SAVs to make sure they have a head start in the SAV market in the future. One of the major hurdles in the way of SAVs making it to the public roads is the lack of confidence of public in the safety aspect of SAVs. In order to assure safety and provide confidence to the public in the safety of SAVs, researchers around the world have used coverage-based testing for Verif…
Institute for Software Integrated Systems Vanderbilt University（范德堡大学软件集成系统研究所）；Raytheon BBN Technologies（雷神BBN技术公司）
Safety Case has become an integral component for safety-certification in various Cyber Physical System domains including automotive, aviation, medical devices, and military. The certification processes for these systems are stringent and require robust safety assurance arguments and substantial evidence backing. Despite the strict requirements, current practices still rely on manual methods that are brittle, do not have a systematic approach or thorough consideration of sound arguments. In addition, stringent certification requirements and ever-increasing system complexity make ad-hoc, manual …
Safety-critical software systems are those whose failure or malfunction could result in casualty and/or serious financial loss. In such systems, safety assurance cases (SACs) are an emerging approach that adopts a proactive strategy to produce structuralized safety justifications and arguments. While SACs are recommended in many software-intensive safety-critical domains, the lack of knowledge regarding the practitioners' perspectives on using SACs hinders effective adoption of this approach. To gain such knowledge, we interviewed nine practitioners and safety experts who focused on safety-cri…
We propose a method for deploying a safety-critical machine-learning component into continuously evolving environments where an increased degree of automation in the engineering process is desired. We associate semantic tags with the safety case argumentation and turn each piece of evidence into a quantitative metric or a logic formula. With proper tool support, the impact can be characterized by a query over the safety argumentation tree to highlight evidence turning invalid. The concept is exemplified using a vision-based emergency braking system of an autonomous guided vehicle for factory a…
The European Machinery Directive and related harmonized standards do consider that software is used to generate safety-relevant behavior of the machinery but do not consider all kinds of software. In particular, software based on machine learning (ML) are not considered for the realization of safety-relevant behavior. This limits the introduction of suitable safety concepts for autonomous mobile robots and other autonomous machinery, which commonly depend on ML-based functions. We investigated this issue and the way safety standards define safety measures to be implemented against software fau…
This report provides an introduction and overview of the Technical Topic Notes (TTNs) produced in the Towards Identifying and closing Gaps in Assurance of autonomous Road vehicleS (Tigars) project. These notes aim to support the development and evaluation of autonomous vehicles. Part 1 addresses: Assurance-overview and issues, Resilience and Safety Requirements, Open Systems Perspective and Formal Verification and Static Analysis of ML Systems. Part 2: Simulation and Dynamic Testing, Defence in Depth and Diversity, Security-Informed Safety Analysis, Standards and Guidelines.
A Verification Methodology for Safety Assurance of Robotic Autonomous Systems
Faculty of Science and Technology, Norwegian University Of Life Sciences (NMBU)（科学与技术学院，挪威生命科学大学）；Department of Mechatronics, University of Agder (UiA)（机械电子系，阿格德大学）；Department of Computer Science, University Of York, UK（计算机科学系，约克大学（英国））
Dealing with Commercial off-the-shelf (COTS) com- ponents is a daily business for avionic system manufacturers. They are necessary ingredients for hardware designs, but are not built in accordance with the avionics consensus standard DO- 254 for Airborne Electronic Hardware (AEH) design. Especially for complex COTS hardware components used in safety critical AEH, like Microcontroller Units (MCUs), additional assurance activities have to be performed. All of them together shall form a convincing confident, that the hardware is safe in its intended operation environment. The focus of DO-254 is o…
Fraunhofer Institute for Experimental Software Engineering IESE（弗劳恩霍夫实验软件工程研究所）
In the future, AI will increasingly find its way into systems that can potentially cause physical harm to humans. For such safety-critical systems, it must be demonstrated that their residual risk does not exceed what is acceptable. This includes, in particular, the AI components that are part of such systems' safety-related functions. Assurance cases are an intensively discussed option today for specifying a sound and comprehensive safety argument to demonstrate a system's safety. In previous work, it has been suggested to argue safety for AI components by structuring assurance cases based on…
This paper presents a general framework and methods for complete programming and checking of distributed algorithms at a high-level, as in pseudocode languages, but precisely specified and directly executable, as in formal specification languages and practical programming languages, respectively. The checking framework, as well as the writing of distributed algorithms and specification of their safety and liveness properties, use DistAlgo, a high-level language for distributed algorithms. We give a complete executable specification of the checking framework, with a complete example algorithm a…
Assurance cases are structured arguments that are commonly used to reason about the safety of a product or service. Currently, there is an ongoing push towards using assurance cases for also cybersecurity, especially in safety-critical domains, like automotive. While the industry is faced with the challenge of defining a sound methodology to build security assurance cases, the state of the art is rather immature. Therefore, we have conducted a thorough investigation of the (external) constraints and (internal) needs that security assurance cases have to satisfy in the context of the automotive…
Probabilistic modelling and safety assurance of an agriculture robot providing light-treatment
Robotics Group, Faculty of Science & Technology, Norwegian University of Life Sciences (NMBU)（挪威生命科学大学机器人小组，科学与技术学院）；Dept. of Computer Science, University of York（约克大学计算机科学系）；Dept. of Mechatronics, Faculty of Engineering and Science, University of Agder (UiA)（阿格德大学工程与科学学院机电系）；Saga Robotics AS（Saga机器人公司）
This report provides an introduction and overview of the Technical Topic Notes (TTNs) produced in the Towards Identifying and closing Gaps in Assurance of autonomous Road vehicleS (Tigars) project. These notes aim to support the development and evaluation of autonomous vehicles. Part 1 addresses: Assurance-overview and issues, Resilience and Safety Requirements, Open Systems Perspective and Formal Verification and Static Analysis of ML Systems. This report is Part 2 and discusses: Simulation and Dynamic Testing, Defence in Depth and Diversity, Security-Informed Safety Analysis, Standards and G…
NHS Digital（NHS数字部）；Kheiron Medical Technologies（凯龙医疗技术公司）；Health Navigator（健康导航公司）；University of York（约克大学）
In recent years, the number of machine learning (ML) technologies gaining regulatory approval for healthcare has increased significantly allowing them to be placed on the market. However, the regulatory frameworks applied to them were originally devised for traditional software, which has largely rule-based behaviour, compared to the data-driven and learnt behaviour of ML. As the frameworks are in the process of reformation, there is a need to proactively assure the safety of ML to prevent patient safety being compromised. The Assurance of Machine Learning for use in Autonomous Systems (AMLAS)…
Security Assurance Cases (SAC) are a form of structured argumentation used to reason about the security properties of a system. After the successful adoption of assurance cases for safety, SACs are getting significant traction in recent years, especially in safety-critical industries (e.g., automotive), where there is an increasing pressure to be compliant with several security standards and regulations. Accordingly, research in the field of SAC has flourished in the past decade, with different approaches being investigated. In an effort to systematize this active field of research, we conduct…
Assurance cases provide an organized and explicit argument for correctness. They can dramatically improve the certification of Scientific Computing Software (SCS). Assurance cases have already been effectively used for safety cases for real time systems. Their advantages for SCS include engaging domain experts, producing only necessary documentation, and providing evidence that can be verified/replicated. This paper illustrates assurance cases for SCS through the correctness case for 3dfim+, an existing Medical Imaging Application (MIA) for analyzing activity in the brain. This example was par…
Recent advances in artificial intelligence and machine learning may soon yield paradigm-shifting benefits for aerospace systems. However, complexity and possible continued on-line learning makes neural network control systems (NNCS) difficult or impossible to certify under the United States Military Airworthiness Certification Criteria defined in MIL-HDBK-516C. Run time assurance (RTA) is a control system architecture designed to maintain safety properties regardless of whether a primary control system is fully verifiable. This work examines how to satisfy compliance with MIL-HDBK-516C while u…
At the Centre for Proton Therapy at the Paul Scherrer Institute cancer patients are treated with a fixed beamline and in two gantries for ocular and non-ocular malignancies, respectively. For the installation of a third gantry a new patient safety system (PaSS) was developed and is sequentially being rolled out to update the existing areas. The aim of PaSS is to interrupt the treatment whenever any sub-system detects a hazardous condition. To ensure correct treatment delivery, this system needs to be thoroughly tested as part of the regular quality assurance (QA) protocols as well as after any…
University of Waterloo（滑铁卢大学）；DENSO CORPORATION（电装公司）
Safety assurance is a central concern for the development and societal acceptance of automated driving (AD) systems. Perception is a key aspect of AD that relies heavily on Machine Learning (ML). Despite the known challenges with the safety assurance of ML-based components, proposals have recently emerged for unit-level safety cases addressing these components. Unfortunately, AD safety cases express safety requirements at the system level and these efforts are missing the critical linking argument needed to integrate safety requirements at the system level with component performance requiremen…
RISE Research Institutes of Sweden（瑞典RISE研究院）；Lund University（隆德大学）；Semcon AB（Semcon公司）；Infotiv AB（Infotiv公司）；QRTECH AB（QRTECH公司）；Combitech AB（Combitech公司）
Integration of Machine Learning (ML) components in critical applications introduces novel challenges for software certification and verification. New safety standards and technical guidelines are under development to support the safety of ML-based systems, e.g., ISO 21448 SOTIF for the automotive domain and the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) framework. SOTIF and AMLAS provide high-level guidance but the details must be chiseled out for each specific case. We initiated a research project with the goal to demonstrate a complete safety case for an ML component…
Dependability assurance of systems embedding machine learning(ML) components---so called learning-enabled systems (LESs)---is a key step for their use in safety-critical applications. In emerging standardization and guidance efforts, there is a growing consensus in the value of using assurance cases for that purpose. This paper develops a quantitative notion of assurance that an LES is dependable, as a core component of its assurance case, also extending our prior work that applied to ML components. Specifically, we characterize LES assurance in the form of assurance measures: a probabilistic …
Harbin Institute of Technology（哈尔滨工业大学）；Nanyang Technological University（南洋理工大学）；Carnegie Mellon University（卡内基梅隆大学）；Optus Macquarie University Cyber Security Hub（Optus麦考瑞大学网络安全中心）；Kyushu University（九州大学）；University of Illinois at Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校）；NVIDIA AI Tech Centre（英伟达人工智能技术中心）
Over the past decades, deep learning (DL) systems have achieved tremendous success and gained great popularity in various applications, such as intelligent machines, image processing, speech processing, and medical diagnostics. Deep neural networks are the key driving force behind its recent success, but still seem to be a magic black box lacking interpretability and understanding. This brings up many open safety and security issues with enormous and urgent demands on rigorous methodologies and engineering practice for quality enhancement. A plethora of studies have shown that the state-of-the…
Arguments about the safety, security, and correctness of a complex system are often made in the form of an assurance case. An assurance case is a structured argument, often represented with a graphical interface, that presents and supports claims about a system's behavior. The argument may combine different kinds of evidence to justify its top level claim. While assurance cases deliver some level of guarantee of a system's correctness, they lack the rigor that proofs from formal methods typically provide. Furthermore, changes in the structure of a model during development may result in inconsi…
