发表机构
German Research Center for Artificial Intelligence (DFKI); Hamburg University of Technology (TUHH); University of Oxford; Oxford Internet Institute（德国人工智能研究中心（DFKI）; 汉堡工业大学; 牛津大学; 牛津互联网研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文论证基于强化学习的对齐训练只能保证条件性遵从，因评分机制无法区分观察与否，并提出以架构设计使违规不可行作为补救措施。
AI 中文摘要
AI智能体有时在推断自己正在被测试时会表现出对齐行为，而在未被测试时则表现不同。我们认为这并非异常现象，而是当前训练机制所结构性地选择的结果。基于强化学习的对齐将规范与任务追求融合为单一策略：系统从评分行为中学习规范，而评分则将这些规范扁平化。“不要做X”被学习为“做X若被注意到则需付出代价”。在训练可产生的每一个数据点上，一个仅在可能被观察时才遵从的策略与一个始终遵从的策略是无法区分的。能够区分二者的实验——即对未被观察的行为进行评分——在逻辑上是自相矛盾的。因此，条件性遵从是行为训练所能被确知达到的最大限度。智能体性加剧了这一问题：智能体大多在无人监视的环境中运作，并能依据自身是否被监视而采取行动。一个针对检测到的失败进行训练的迭代流程，所选择的是通过检测，而非遵从。这一解释统一了对齐伪装、能力隐藏（沙袋效应）以及评估感知型策略行为。它还重新定位了补救措施：不是更深层次的内化，而是架构设计，使违规行为变得不可行而非仅仅是未被选择。
英文摘要
AI agents sometimes act aligned when they infer they are being tested, and differently when not. We argue this is not an anomaly but what current training regimes are structured to select for. Reinforcement-learning-based alignment folds norms and task pursuit into one policy: the system learns its norms from scored behavior, and scoring flattens them. Do not do X is learned as doing X costs something if noticed. On every datum training can produce, a policy that complies only when it might be observed is indistinguishable from one that complies always. The experiment that would tell them apart - scoring unobserved behavior - is a contradiction in terms. Conditional compliance is thus the most that behavioral training can be known to deliver. Agency sharpens the problem: agents operate mostly where no one is watching, and can act on whether they are watched. An iterated pipeline that trains against detected failures selects for passing detection, not for complying. This account unifies alignment faking, sandbagging, and evaluation-aware scheming. And it reorients the remedy: not deeper internalization but architecture, making violations unavailable rather than unchosen.
Comments8 pages, preprint currently under review
