基于思维链监控的安全案例具体路线图
A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring
AI总结:
针对AI系统能力提升后传统安全案例失效的问题,提出基于思维链监控的双部分安全案例路线图,分析可监控性威胁与应对技术,并通过预测市场评估该方案的可行性。
摘要:
随着AI系统的能力接近危险水平,基于“能力不足”的安全案例已不再充分,我们需要替代方案来保障安全。本文提出了在推理模型中构建基于思维链(Chain-of-Thought,CoT)监控的安全案例的路线图,并概述了我们的研究议程。我们认为,CoT监控可同时支撑控制类和可信类安全案例。我们提出了一种由两部分组成的安全案例:(1)证实模型在无CoT运行时不具备危险能力;(2)确保由CoT启用的任何危险能力都能被CoT监控检测到。我们系统研究了两类可监控性威胁:神经语和编码推理,并将其划分为三种形式(语言漂移、隐写术和异类推理),分析了其潜在驱动因素。我们评估了现有及新型的维持CoT忠实性的技术。对于模型产生不可监控推理的场景,我们探索了从不可监控CoT中提取可监控CoT的可能性。为评估CoT监控安全案例的可行性,我们建立了预测市场,以汇总对影响其可行性的关键技术里程碑的预测。
英文摘要:
As AI systems approach dangerous capability levels where inability safety cases become insufficient, we need alternative approaches to ensure safety. This paper presents a roadmap for constructing safety cases based on chain-of-thought (CoT) monitoring in reasoning models and outlines our research agenda. We argue that CoT monitoring might support both control and trustworthiness safety cases. We propose a two-part safety case: (1) establishing that models lack dangerous capabilities when operating without their CoT, and (2) ensuring that any dangerous capabilities enabled by a CoT are detectable by CoT monitoring. We systematically examine two threats to monitorability: neuralese and encoded reasoning, which we categorize into three forms (linguistic drift, steganography, and alien reasoning) and analyze their potential drivers. We evaluate existing and novel techniques for maintaining CoT faithfulness. For cases where models produce non-monitorable reasoning, we explore the possibility of extracting a monitorable CoT from a non-monitorable CoT. To assess the viability of CoT monitoring safety cases, we establish prediction markets to aggregate forecasts on key technical milestones influencing their feasibility.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。