Motivated by practical needs of experimentation and policy learning in online platforms, we study the problem of safe data collection. Specifically, our goal is to develop a logging policy that efficiently explores different actions to elicit information while achieving competitive reward with a baseline production policy. We first show that a common practice of mixing the production policy with randomized exploration, despite being safe, is sub-optimal in maximizing information gain. Then, we propose a safe optimal logging policy via a novel water-filling technique for the case when no side information about the actions' expected reward is available. We improve upon this design by considering side information and also extend our approaches to the linear contextual model to account for a large number of actions.
Along the way, we analyze how our data logging policies impact errors in off(line)-policy learning and empirically validate the benefit of our design by conducting extensive numerical experiments with synthetic and MNIST datasets. To further demonstrate the generality of our approach, we also consider the safe online learning setting. By adaptively applying our techniques, we develop the Safe Phased-Elimination (SafePE) algorithm that can achieve optimal regret bound with only logarithmic number of policy updates.
FOSP: Fine-tuning Offline Safe Policy through World Models
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);College of Design and Engineering, National University of Singapore(新加坡国立大学设计与工程学院)
CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learning
Tongji University(同济大学);JD Explore Academy(京东探索研究院);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区网络科学与技术学院);Berkeley AI Research Lab, University of California, Berkeley(加州大学伯克利分校伯克利人工智能研究实验室);Technical University of Munich(慕尼黑工业大学)
Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
Radboud University Nijmegen The Netherlands;Ruhr University\,\&\,Radboud University Bochum Germany;Radboud University;Ruhr University\,\&\,Radboud University
Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);School of Vehicle and Mobility, Tsinghua University(清华大学车辆与运载学院);Department of Computer Science, The University of Hong Kong(香港大学计算机科学系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Polixir Technologies Nanjing, China(南京Polixir科技有限公司);Tencent Game AI Center Shenzhen, China(腾讯深圳游戏AI中心)
Safe Evaluation For Offline Learning: Are We Ready To Deploy?
University of Alberta(阿尔伯塔大学);University of Wisconsin – Madison(威斯康星大学麦迪逊分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Sony AI(索尼人工智能研究院);Alberta Machine Intelligence Institute(阿尔伯塔机器智能研究院)
Individual Control Barrier Functions-Guided Diffusion Model for Safe Offline Multi-Agent Reinforcement Learning
Department of Electrical Engineering and Automation, Aalto University(阿尔托大学电气工程与自动化系);School of Computing and Data Science, Xiamen University Malaysia(厦门大学马来西亚分校计算与数据科学学院);Department of Computer Science, University of Toronto(多伦多大学计算机科学系)
Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
School of Electrical, Computer and Energy Engineering(电气、计算机与能源工程学院);Arizona State University(亚利桑那州立大学);School of Electrical Engineering and Computer Science(电气工程与计算机科学学院);Washington State University(华盛顿州立大学);Department of Computer Science(计算机科学系);University of Houston(休斯顿大学)
CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning
Alibaba-NTU Singapore Joint Research Institute(阿里巴巴-南洋理工大学新加坡联合研究院);Nanyang Technological University(南洋理工大学);Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly(南洋理工大学-不列颠哥伦比亚大学活跃养老卓越研究中心);Centre for Frontier AI Research(前沿人工智能研究中心);Agency for Science, Technology and Research(新加坡科技研究局)
DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving
College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院);Maynooth International College of Engineering, Fuzhou University(福州大学梅努斯国际工程学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
Department of Chemical Engineering, Imperial College London(帝国理工学院化学工程系);Shell Information Technology International BV(壳牌信息科技国际有限公司);Shell Global Solutions International BV(壳牌全球解决方案国际有限公司)