上游与下游 AI 安全:两者同在一条河上吗?
Upstream and Downstream AI Safety: Both on the Same River?
AI总结:
本文对比上游前沿 AI 安全与下游传统系统安全框架,探讨二者概念与能力分析相互迁移的路径、实例及实现协同面临的挑战。
摘要:
传统安全工程在系统的使用情境中评估系统,例如自动驾驶车辆(包括使用 AI 的车辆)的运行设计域(道路布局、限速、天气等)。我们将此称为下游安全。相比之下,关于前沿 AI 安全性的工作,例如可进一步训练以用于下游任务的大型语言模型,通常考虑超出具体应用情境的因素,如模型逃避人类控制的能力,或生成有害内容(例如如何制造炸弹)的能力。我们将此称为上游安全。我们概述了上游和下游安全框架的特征,随后探讨广义 AI 安全共同体在多大程度上能够受益于这些框架之间的协同作用。例如,来自下游安全的共模故障等概念能否用于帮助评估 AI guardrails(防护栏)的强度?此外,对前沿 AI 能力和局限的理解能否用于为下游安全分析提供信息,例如在 LLM 被微调以计算自主船舶航行计划的情形中?本文指出了一些值得探索的有前景路径,并概述了在上游与下游安全框架之间实现协同或汇合所面临的若干挑战。
英文摘要:
Traditional safety engineering assesses systems in their context of use, e.g. the operational design domain (road layout, speed limits, weather, etc.) for self-driving vehicles (including those using AI). We refer to this as downstream safety. In contrast, work on safety of frontier AI, e.g. large language models which can be further trained for downstream tasks, typically considers factors that are beyond specific application contexts, such as the ability of the model to evade human control, or to produce harmful content, e.g. how to make bombs. We refer to this as upstream safety. We outline the characteristics of both upstream and downstream safety frameworks then explore the extent to which the broad AI safety community can benefit from synergies between these frameworks. For example, can concepts such as common mode failures from downstream safety be used to help assess the strength of AI guardrails? Further, can the understanding of the capabilities and limitations of frontier AI be used to inform downstream safety analysis, e.g. where LLMs are fine-tuned to calculate voyage plans for autonomous vessels? The paper identifies some promising avenues to explore and outlines some challenges in achieving synergy, or a confluence, between upstream and downstream safety frameworks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。