AI保障可能在何处出错：关键系统工程化的初步教训
Where AI Assurance Might Go Wrong: Initial lessons from engineering of critical systems
AI总结：
本文从传统关键系统安全工程经验出发，分析AI安全框架的不足，提出基于Assurance 2.0的保障案例方法，以支持对系统与决策关键性的评估。
摘要：
我们基于在关乎社会的重要系统上开展系统与软件保障及评估工作的经验，总结传统关键系统（如飞机飞行控制）中安全工程是如何实施的。我们分析这种关键系统视角如何支持AI安全框架的开发与实施。我们从系统工程、安全与风险分析以及决策分析与支持三个方面呈现该分析。我们考虑四个关键问题：系统是什么？它必须有多好？关键性对系统开发有何影响？我们应该在多大程度上信任它？我们识别出值得进一步讨论的主题。我们尤其担忧：系统边界不够宽泛，风险的可容忍性与性质阐述不够充分，以及保障方法缺乏能够使行为得到充分保障的理论。我们倡导使用基于Assurance 2.0的保障案例来支持决策，在决策中同时评估决策的关键性和系统的关键性。我们指出关键系统与日常系统所需置信度存在数量级差异，以及日常技术如何在严谨性上无法扩展。最后，我们将研究结果详细映射到FAISC组织者提出的两个问题，并指出关键系统工程是通过开放和多元的讨论演进的。我们希望这里识别的主题能够支持FAISC之后的对话。
英文摘要：
We draw on our experience working on system and software assurance and evaluation for systems important to society to summarise how safety engineering is performed in traditional critical systems, such as aircraft flight control. We analyse how this critical systems perspective might support the development and implementation of AI Safety Frameworks. We present the analysis in terms of: system engineering, safety and risk analysis, and decision analysis and support. We consider four key questions: What is the system? How good does it have to be? What is the impact of criticality on system development? and How much should we trust it? We identify topics worthy of further discussion. In particular, we are concerned that system boundaries are not broad enough, that the tolerability and nature of the risks are not sufficiently elaborated, and that the assurance methods lack theories that would allow behaviours to be adequately assured. We advocate the use of assurance cases based on Assurance 2.0 to support decision making in which the criticality of the decision as well as the criticality of the system are evaluated. We point out the orders of magnitude difference in confidence needed in critical rather than everyday systems and how everyday techniques do not scale in rigour. Finally we map our findings in detail to two of the questions posed by the FAISC organisers and we note that the engineering of critical systems has evolved through open and diverse discussion. We hope that topics identified here will support the post-FAISC dialogues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
