发表机构
Naval Postgraduate School; Google Research; University of California, San Diego; University of Michigan, Ann Arbor（海军研究生院; 谷歌研究院; 加州大学圣地亚哥分校; 密歇根大学安娜堡分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究从切尔诺贝利等社会技术灾难中汲取未吸取的教训，探讨其对人工智能的启示，指出人工智能开发和使用可在组织风险处理、需求责任追溯及整体安全方法等方面受益，以避免类似灾难在现代系统中重演。
AI 中文摘要
随着自动化决策和数据驱动技术在社会中广泛应用并用于管理重大结果，理解技术在实际中的能力、局限性和相关风险需要分析完整的社会技术系统。对高度复杂系统中的风险进行社会技术分析为人工智能系统的设计和评估提供了清晰的教训，超越了对可靠或‘负责任设计’组件的技术关注，以在系统层面理解风险。人为灾难已被研究数十年，如切尔诺贝利、三里岛、福岛第一核电站、博帕尔、挑战者号灾难等。常见误解是这些事件是罕见事故，源于复杂系统中固有的不可预见的相互作用。仔细研究发现，风险和危害事先是已知的，但由于社会结构、政治和经济因素而未采取行动。我们概述了人工智能的开发和使用可以从吸取这些未吸取教训中受益的几个领域：在组织层面改善风险感知、沟通和分析；需求和责任的可追溯性；以及将社会和组织动态作为一级工程关注点的责任和安全的整体方法。对于每个领域，我们提供具体的未吸取教训，并举例说明它们如何导致先前事故中的失败，以及这些教训在现代计算系统，特别是人工智能中仍未被吸取的例子。
英文摘要
As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, and attendant risks in context requires analysis of full sociotechnical systems. Sociotechnical analysis of risks in highly complex systems provides clear lessons for the design and evaluation of AI systems, transcending a technical focus on reliable or "responsibly designed" components to understand risks at a systems level. Human-made catastrophes have been studied for decades because of the severity of these events: consider Chernobyl, Three Mile Island, Fukushima-Daiichi, Bhopal, the Challenger disaster. A common misconception is that these kinds of events are freak accidents, resulting from the inherently unforeseeable interactions in complex systems. Closer examination reveals that the risks and hazards were well-known beforehand but not acted upon due to social structural, political and economic factors. We outline several areas where the development and use of AI can benefit from learning these unlearned lessons: improved risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety that include social and organizational dynamics as first-order engineering concerns. For each area, we offer concrete unlearned lessons and exemplify how they led to failure in prior accidents as well as examples of how these lessons remain unlearned for modern computing systems, particularly AI.
CommentsAccepted to the Harvard Data Science Review, Volume 8(3), Summer 2026
Journal refHarvard Data Science Review, Volume 8(3), Summer 2026
