Safe reinforcement learning (RL) studies problems where an intelligent agent has to not only maximize reward but also avoid exploring unsafe areas. In this study, we propose CUP, a novel policy optimization method based on Constrained Update Projection framework that enjoys rigorous safety guarantee. Central to our CUP development is the newly proposed surrogate functions along with the performance bound. Compared to previous safe RL methods, CUP enjoys the benefits of 1) CUP generalizes the surrogate functions to generalized advantage estimator (GAE), leading to strong empirical performance. 2) CUP unifies performance bounds, providing a better understanding and interpretability for some existing algorithms; 3) CUP provides a non-convex implementation via only first-order optimizers, which does not require any strong approximation on the convexity of the objectives. To validate our CUP method, we compared CUP against a comprehensive list of safe RL baselines on a wide range of tasks. Experiments show the effectiveness of CUP both in terms of reward and safety constraint satisfaction. We have opened the source code of CUP at this link https://github.com/zmsn-2077/ CUP-safe-rl.
COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection
Department of Cognitive Robotics, Delft University of Technology（代尔夫特理工大学认知机器人系）；Department of Intelligent Systems, Delft University of Technology（代尔夫特理工大学智能系统系）；School of Transportation, Southeast University（东南大学交通学院）
Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；The State Key Lab of Brain-Machine Intelligence, Zhejiang University（浙江大学脑机智能国家重点实验室）；Center for AI Safety and Governance, Peking University（北京大学人工智能安全与治理中心）
SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
Université de Montréal（蒙特利尔大学）；Mila - Québec AI Institute（蒙特利尔人工智能研究所）；Université Laval（拉瓦尔大学）；IID - Institut Intelligence et Données（智能与数据研究所）；LawZero；CIFAR AI Chair（CIFAR人工智能主席）
Brain-SAD: A Brain-Inspired Safe Autonomous Driving Control Framework with Dynamic Fear-Oriented Constraint on Dual-Policy
Nanjing University of Information Science & Technology（南京信息工程大学）；University of Electronic Science and Technology of China（电子科技大学）；Jiangsu Ocean University（江苏海洋大学）
Feasible Policy Iteration for Safe Reinforcement Learning
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与出行学院）；College of Artificial Intelligence, Tsinghua University（清华大学人工智能学院）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）
Multi-Level Safety Continual Projection for Fine-Tuned Large Language Models without Retraining
Beijing Key Laboratory of Safe AI and Superalignment（北京安全人工智能与超对齐重点实验室）；Beijing Institute of AI Safety and Governance（北京人工智能安全与治理研究院）；Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences（脑启发认知人工智能实验室，中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Long-term AI（长期人工智能）
SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selection
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学计算机科学与工程系）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Platform and Content Group, Tencent（腾讯平台与内容组）；Chinese Medicine Guangdong Laboratory（广东中医实验室）
Department of Computing（计算系）；Imperial College London（帝国理工学院伦敦分校）；School of Computer and Comm. Sciences (IC)（计算机与通信科学系）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning
Purdue University（普渡大学）；Brown University（布朗大学）；Massachusetts Institute of Technology（麻省理工学院）；University of California, Irvine（加州大学 Irvine 分校）；Utah State University（犹他州立大学）；Harvard University（哈佛大学）
Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection
Wuhan University（武汉大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；China University of Mining and Technology（中国矿业大学）
Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
School of Life Sciences, IDG/McGovern Institute for Brain Research（生命科学学院，IDG/麦戈文脑科学研究所）；Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center（计算机科学与技术系，人工智能研究所，清华大学-博世联合机器学习中心，THBI实验室，BNRist中心）；Tsinghua University, Beijing, China（清华大学，北京，中国）
Safe and Reliable Diffusion Models via Subspace Projection
Faculty of Data Science, City University of Macau（澳门大学数据科学学院）；School of Computer Science, University of Queensland（昆士兰大学计算机科学学院）；Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院）山东省计算中心）
Ocean University of China（中国海洋大学）；State Key Laboratory of Integrated Services Networks (Xidian University)（综合业务网理论及关键技术国家重点实验室（西安电子科技大学））；Southeast University（东南大学）；Purple Mountain Laboratories（紫金山实验室）
