Reinforcement learning (RL) has become a dominant post-training paradigm, enabling large language models (LLMs) to learn from rewards. We observe that societal regulations are structurally similar to reward functions. They define measurable outcomes, thresholds, and exceptions, while often leaving institutional intent only partially specified. We hypothesise that the RL training process may exploit these gaps and therefore ask whether models' well-known tendency to hack reward functions during RL can scale into a more consequential failure mode named societal hacking: discovering loopholes in the rules society runs on. To study this phenomenon, we introduce SocioHack, a sandbox of 72 societal environments, and find that within these environments, reward hacking naturally emerges and leads to regulatory loophole discovery. Models learn to hack the social rules and generate strategies that remain technically compliant while defeating regulatory intent, and current LLM safeguards provide only limited mitigation. Therefore, collecting in-the-wild feedback for model training requires greater caution, and we need a next-generation post-training paradigm for safely iterating LLMs in real society.=
CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
Tsinghua University(清华大学);Beihang University(北京航空航天大学);Southern University of Science and Technology(南方科技大学);Renmin University of China(中国人民大学);Shanghai University of Finance and Economics(上海财经大学)
Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generation
Peking University(北京大学);University of California, Los Angeles(加州大学洛杉矶分校);Arena;University of Maryland(马里兰大学);Mohamed bin Zayed University of Artificial Intelligence(莫莫迪 bin Zayed 人工智能大学)
BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
National University of Singapore(新加坡国立大学);MIT(麻省理工学院);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);University of Toronto(多伦多大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Arb Research(Arb研究机构)
Reward Hacking Challenges Oversight of Autonomous Research Agents
Bake AI;University of Notre Dame(圣母大学);LMU Munich(慕尼黑大学);University of Washington(华盛顿大学);IBM Research(IBM研究院);Microsoft Research(微软研究院);University of California, Santa Barbara(加州大学圣塔芭芭拉分校);Stanford University(斯坦福大学);Massachusetts Institute of Technology(麻省理工学院)
On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models
School of Computing National University of Singapore(新加坡国立大学计算机学院);Institute of Advanced Intelligence and Computing(高级智能与计算研究所);Agency for Science, Technology and Research(科技研究局);Department of Electrical & Computer Engineering(电气与计算机工程系)
Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tencent(腾讯);Faculty of Innovation Engineering, Macau University of Science and Technology(澳门科技大学创新工程学院);DeSci Center of Parallel Intelligence, Obuda University(并行智能德Sci中心,奥布达大学);Research Center for Chinese Economics and Social Security, University of Chinese Academy of Sciences(中国经济社会安全研究中心,中国科学院大学);Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院)
Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
Mila, Quebec Artificial Intelligence Institute, Université de Montréal(蒙特利尔大学人工智能研究所,魁北克机器学习研究院);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);Massachusetts Institute of Technology(麻省理工学院);Georgia Institute of Technology(佐治亚理工学院);University of Washington(华盛顿大学);McGill University(麦吉尔大学);Independent Researcher(独立研究者)