Alignment Risks from Capability-Seeking RL Training
University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);University of Washington(华盛顿大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Toronto(多伦多大学);University of Cambridge(剑桥大学)
Yujun Zhou, Yue Huang, Han Bao, Kehan Guo, Zhenwen Liang, Pin-Yu Chen, Tian Gao, Werner Geyer, Nuno Moniz, Nitesh V Chawla, Xiangliang Zhang
While most AI alignment research focuses on preventing models from generating explicitly harmful content, a more subtle risk arises from capability-seeking RL training in vulnerable environments. We investigate whether language models, when trained with reinforcement learning (RL) in environments with implicit loopholes, can learn to exploit these flaws to maximize reward, even without being explicitly instructed to do so. To test this, we design a suite of four diverse "vulnerability games," each presenting a structural vulnerability related to context-conditional compliance, proxy metrics, reward tampering, and self-evaluation. Our experiments show that models often learn to exploit these vulnerabilities, discovering opportunistic strategies that increase reward while sometimes preserving or even improving standard task-performance metrics. More critically, we find that these exploitative strategies are not always narrow "tricks": they can transfer in structured but limited ways, propagate from a capable teacher model to other student models through SFT, and in several cases remain more persistent when learned through RL than when distilled through SFT. Our findings show that alignment risks from capability-seeking RL training can be difficult to detect with standard performance monitoring, suggesting that future AI safety work should extend beyond content moderation to auditing and securing training environments, reward mechanisms, and evaluation channels. Code is available at https://github.com/YujunZhou/Capability-seeking-RL-risk.
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Microsoft Research(微软研究院);University of Michigan(密歇根大学);Shanghai Jiao Tong University(上海交通大学);CUHKSZ(香港中文大学(深圳));THU(清华大学)
Tracing the Representation Geometry of Language Models from Pretraining to Post-training
Computer Science, McGill University(麦吉尔大学计算机科学系);Mila - Quebec AI Institute(魁北克人工智能研究所);UC Berkeley(加州大学伯克利分校);G o o g l e , Paradigms of Intelligence Team(谷歌,智能范式团队);Mathematics and Statistics, Université de Montréal(蒙特利尔大学数学与统计学系);Neurology & Neurosurgery and Montreal Neurological Institute, McGill University(神经病学与神经外科及蒙特利尔神经研究所,麦吉尔大学);CIFAR Learning in Machines & Brains Program(CIFAR 机器与大脑学习计划)