LLM Agent 中阴谋行为的宪法黑盒监控
Constitutional Black-Box Monitoring for Scheming in LLM Agents
AI总结：
研究使用基于宪法黑盒的监控器，通过仅观察外部输入和输出检测LLM Agent的阴谋行为，并在合成数据上优化后泛化到更真实环境。
摘要：
在自主环境中安全部署大型语言模型（LLM）Agent需要可靠的监督机制。一个核心挑战是检测阴谋行为，即Agent暗中追求不一致的目标。缓解此类风险的一种方法是基于LLM的监控：使用语言模型检查Agent行为中的可疑动作。我们研究宪法黑盒监控器：仅利用外部可观测的输入和输出检测阴谋行为的提示分类器，并在从自然语言行为规范生成的合成数据上优化。我们引入两个生成合成Agent轨迹的流水线：STRIDE（迭代精炼）和Gloom（Agent-环境模拟），各生成1000个样本。通过提示扫描、人工精炼和自动提示优化，我们在这些数据集上优化前沿LLM监控器，并在ControlArena（一套Agent在更现实环境中运行的接地环境）中的7500个保留轨迹上评估性能。结果表明，仅基于合成数据选择的监控器可以泛化到更现实的环境，捕获有意义的阴谋信号。然而，我们发现性能在我们的设置中迅速饱和，简单的提示扫描匹配了更广泛优化的结果。超越这一限制不会带来进一步改进，反而导致过拟合。
英文摘要：
Safe deployment of Large Language Model (LLM) agents in autonomous settings requires reliable oversight mechanisms. A central challenge is detecting scheming, where agents covertly pursue misaligned goals. One approach to mitigating such risks is LLM-based monitoring: using language models to examine agent behaviors for suspicious actions. We study constitutional black-box monitors: prompted classifiers that detect scheming using only externally observable inputs and outputs, optimized on synthetic data generated from natural-language behavior specifications. We introduce two pipelines for generating synthetic agent trajectories, STRIDE (iterative refinement) and Gloom (agent-environment simulation), from which we generate 1,000 samples each. We optimize frontier LLM monitors on these datasets via prompt sweeps, human refinement, and automated prompt optimization, and evaluate performance on 7,500 held-out trajectories from ControlArena, a suite of grounded environments where agents operate in more realistic contexts. Our results demonstrate that monitors selected purely on synthetic data can generalize to more realistic environments, capturing a meaningful scheming signal. However, we find that performance saturates quickly in our setting, with simple prompt sweeps matching the results of more extensive optimization. Pushing beyond this limit yields no further improvements and instead leads to overfitting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
