发表机构
Columbia University; Dragoman; State Street; Future Impact Group（哥伦比亚大学; Dragoman; 道富集团; 未来影响集团）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文借鉴历史技术灾难案例，指出AI开发组织面临类似的结构性风险，现有安全措施可能失效，需在灾难前阶段识别并干预这些动态。
AI 中文摘要
关于人工智能风险的研究主要聚焦于能力风险：即系统变得过于强大、过于自主或与人类价值观严重偏离的危险。然而，对于组织层面的关注却少得多——即构建这些系统的机构本身是否倾向于向失败漂移。本文认为，它们确实如此。无论AI系统变得多么强大，构建它们的组织都面临着与过去重大技术灾难之前相同的结构性动态。通过借鉴挑战者号航天飞机、三哩岛核事故和波音737 MAX坠机事件的案例研究，本文识别了每次失败之前共同的结构性机制，并将其映射到当代AI开发中。研究结果表明，现有的安全基础设施可能提供的保护比表面看起来的要少，因为组织可以在完全合规的情况下完成安全流程，却仍然产生灾难性后果。AI开发的灾难前阶段仍在进行中；本文的目的是在这些动态仍可被中断时使其变得清晰可辨。
英文摘要
Work on the risks of artificial intelligence has focused predominantly on capability risk: the danger that systems become too powerful, too autonomous, or too misaligned with human values. Far less attention has been paid to the organizational level---to whether the institutions building these systems are themselves predisposed to drift toward failure. This paper argues that they are. Regardless of how capable AI systems become, the organizations building them face the same structural dynamics that preceded past major technological disasters. Drawing on case studies of the Space Shuttle Challenger, the Three Mile Island accident, and the Boeing 737 MAX crashes, this paper identifies the common structural mechanisms preceding each failure and maps them onto contemporary AI development. The findings suggest that existing safety infrastructure may provide less protection than it appears, as organizations can complete safety processes in full compliance and still produce catastrophic outcomes. The pre-disaster period of AI development is still underway; the purpose of this paper is to make these dynamics legible while they can still be interrupted.
