量化复杂社会技术系统中采用AI带来的系统层面危害

中文摘要 AI 辅助

人工智能（AI）正日益融入包括关键国家基础设施（CNI）在内的复杂社会技术系统，其危害源于技术、人类与组织要素之间的相互作用。然而当前的AI评估仍以模型为中心，几乎无法洞察观测到的行为如何转化为系统层面的风险。我们提出了一个将结构化危害分析、组件级测试与概率系统建模相结合的框架，以填补这一空白。该框架提供了从模型行为到系统层面结果的可追溯路径，使从业者能够解答AI故障的“后果如何”问题，量化其系统影响，并迈向复杂系统中AI的基于证据的前瞻性治理。以英国实时全额结算（RTGS）系统作为示例案例，我们运用系统理论过程分析（STPA）推导AI驱动的损失场景，并将基于大语言模型（LLM）的交易对抗操纵作为此类损失场景之一进行研究。组件级实验显示，简单的对抗性输入会引发可测量的行为变化，当AI建议被遵循时，这些变化在我们用于金融传染模型的组件到系统映射下，会改变系统韧性，增加银行倒闭数量，并降低冲击引发级联中断的阈值，尤其在AI被广泛采用或垄断的情况下。

英文摘要

Artificial Intelligence (AI) is increasingly integrated into complex sociotechnical systems, including Critical National Infrastructure (CNI), where harms emerge from interactions between technical, human, and organisational elements. Yet current AI evaluation remains model-centric, offering little insight into how observed behaviours might translate into system-level risk. We propose a framework that links structured hazard analysis, component-level testing, and probabilistic system modelling to bridge this gap. By providing a traceable pathway from model behaviour to system-level outcomes, the framework enables practitioners to answer the "so what?" of AI failures, quantify their systemic impact, and move toward evidence-based and anticipatory governance of AI in complex systems. Applied to the UK's Real Time Gross Settlement (RTGS) system as an illustrative worked example, we derive AI-driven loss scenarios using Systems Theoretic Process Analysis (STPA) and examine adversarial manipulation of LLM-based trading as one such loss scenario. Component-level experiments show that simple adversarial inputs induce measurable behavioural shifts where AI recommendations are followed. Under the component-to-system mapping used here for a financial contagion model, these shifts alter system resilience, increasing bank failures and lowering the threshold at which shocks lead to cascading disruption, particularly under widespread or monopolistic AI adoption.

发表机构

• Advai Ltd（Advai有限公司）
• UK National Cyber Security Centre（英国国家网络安全中心）

机构由 AI 辅助整理，请以论文原文为准。