arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.07974cs.SEcs.AIcs.LG
收藏

从危险识别到控制器设计:面向ML驱动系统的主动式与LLM支持的安全工程

From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems

Carnegie Mellon University(卡内基梅隆大学)

Yining Hong, Christopher S. Timperley, Christian Kästner

AI总结:

本文提倡将危险分析整合到ML驱动软件产品开发中,提出利用大语言模型在人类监督下部分自动化修改后的STPA过程,以解决传统方法对专家依赖强、耗时费力难以集成到实际工作流的问题。

摘要:

机器学习(ML)组件正日益集成到软件产品中,然而其复杂性与固有的不确定性往往导致对个人及整个社会产生意外且危险的后果。尽管存在这些风险,从业者很少采用主动方法在危险发生前进行预测和缓解。传统的安全工程方法,如失效模式与影响分析(FMEA)和系统理论过程分析(STPA),提供了早期风险识别的系统化框架,却很少被采用。本文提倡将危险分析整合到任何ML驱动软件产品的开发中,并呼吁提供更多支持使开发者能够使用该过程。通过使用大语言模型(LLM)在关键步骤由人类监督的情况下部分自动化修改后的STPA过程,我们期望解决两个关键挑战:对经验丰富的安全工程专家的严重依赖,以及传统危险分析耗时且劳动密集的性质,这通常阻碍了其整合到实际开发工作流中。我们通过一个贯穿全文的示例说明了我们的方法,证明许多看似未预料到的问题实际上是可以被预见的。

英文摘要:

Machine learning (ML) components are increasingly integrated into software products, yet their complexity and inherent uncertainty often lead to unintended and hazardous consequences, both for individuals and society at large. Despite these risks, practitioners seldom adopt proactive approaches to anticipate and mitigate hazards before they occur. Traditional safety engineering approaches, such as Failure Mode and Effects Analysis (FMEA) and System Theoretic Process Analysis (STPA), offer systematic frameworks for early risk identification but are rarely adopted. This position paper advocates for integrating hazard analysis into the development of any ML-powered software product and calls for greater support to make this process accessible to developers. By using large language models (LLMs) to partially automate a modified STPA process with human oversight at critical steps, we expect to address two key challenges: the heavy dependency on highly experienced safety engineering experts, and the time-consuming, labor-intensive nature of traditional hazard analysis, which often impedes its integration into real-world development workflows. We illustrate our approach with a running example, demonstrating that many seemingly unanticipated issues can, in fact, be anticipated.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑