AI systems that output their reasoning in natural language offer an opportunity for safety -- we can \emph{monitor} their chain of thought (CoT) for undesirable reasoning, such as the pursuit of harmful objectives. However, the extent to which CoT faithfully reflects the underlying reasoning process, and hence the extent to which it can be usefully monitored, may be influenced by certain aspects of training. We investigate how different \emph{training incentives}, applied to a reasoning model, affect its monitorability. We introduce a novel methodology for measuring monitorability according to whether a monitor can predict a key latent variable using the model's reasoning. When controlling for accuracy, we do not find evidence for consistent effects from commonly used incentives (length penalties and KL regularisation), but we find that adversarial optimisation (penalising monitor accuracy) degrades monitor performance, while direct optimisation for monitorability does not reliably lead to improvements. Our code is available at https://github.com/QiyaoWei/reasoning-under-pressure.
Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
UK AI Security Institute(英国人工智能安全研究所);Apollo Research(阿波罗研究);METR;University of Montreal(蒙特利尔大学);Mila;Anthropic;OpenAI(开放人工智能研究所);Google DeepMind(谷歌DeepMind);Truthful AI;UC Berkeley(伯克利大学);Center for AI Safety(人工智能安全中心);AI Futures Project(人工智能未来项目);Amazon(亚马逊);Scale AI;Magic;Meta;Redwood Research(红木研究)
Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
ML Alignment and Theory Scholars(机器学习对齐与理论学者计划);University of Delaware(特拉华大学);AI Safety Argentina(阿根廷AI安全组织);Buenos Aires AI Safety Hub(布宜诺斯艾利斯AI安全中心);University of Buenos Aires(布宜诺斯艾利斯大学);Center for AI Safety(AI安全中心)
How malicious AI swarms can threaten democracy: The fusion of agentic AI and LLMs marks a new frontier in information warfare
Department of Sustainable Communication Technologies, SINTEF Digital(可持续通信技术系,SINTEF数字);Max Planck Institute for Security and Privacy(安全与隐私研究所);Department of Mathematics, City St George’s University of London(数学系,圣乔治大学);Macrostrategy Research Initiative(战略研究计划);Human Nature Lab, Yale University(人性实验室,耶鲁大学);Department of Politics and Public Administration, University of Konstanz(政治与公共管理系,康斯坦茨大学);Harvard Business School, Harvard University(哈佛商学院,哈佛大学);Department of Psychology, University of Cambridge(心理学系,剑桥大学);Department of Computer Science, University of British Columbia(计算机科学系,不列颠哥伦比亚大学);Department of Human Centered Design & Engineering, University of Washington(以人为本设计与工程系,华盛顿大学);Department of Psychology, New York University(心理学系,纽约大学);Observatory on Social Media and Luddy School of Informatics, Computing, and Engineering, Indiana University(社交媒体观察所和信息、计算与工程学院,印第安纳大学)