We consider the problem of reinforcement learning under safety requirements, in which an agent is trained to complete a given task, typically formalized as the maximization of a reward signal over time, while concurrently avoiding undesirable actions or states, associated to lower rewards, or penalties. The construction and balancing of different reward components can be difficult in the presence of multiple objectives, yet is crucial for producing a satisfying policy. For example, in reaching a target while avoiding obstacles, low collision penalties can lead to reckless movements while high penalties can discourage exploration. To circumvent this limitation, we examine the effect of past actions in terms of safety to estimate which are acceptable or should be avoided in the future. We then actively reshape the action space of the agent during reinforcement learning, so that reward-driven exploration is constrained within safety limits. We propose an algorithm enabling the learning of such safety constraints in parallel with reinforcement learning and demonstrate its effectiveness in terms of both task completion and training time.
Learning to Recover: Dynamic Reward Shaping with Wheel-Leg Coordination for Fallen Robots
Humanoids and Human-Centered Mechatronics (HHCM), Istituto Italiano di Tecnologia（人类oids与人本机电系统（HHCM），意大利技术研究所）；Ph.D. Program of National Interest in Robotics and Intelligent Machines (DRIM), University of Genova（机器人与智能机器国家利益博士项目（DRIM），热那亚大学）；Department of Computer Science, University College London（计算机科学系，伦敦大学学院）
Automatic Reward Shaping from Confounded Offline Data
Causal AI Lab, Columbia University, New York, USA（因果AI实验室，哥伦比亚大学，纽约，美国）；Department of Electrical Engineering and Computer Science, Syracuse University, New York, USA（电气工程与计算机科学系， Syracuse大学，纽约，美国）
Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
Dept. of Computer Science, Stanford University（斯坦福大学计算机科学系）；Dept. of Electrical Engineering and Computer Science, University of California, Berkeley（加州大学伯克利分校电气工程与计算机科学系）；Google AI（谷歌人工智能（Google AI））
Learning to Trust Experience: A Monitor-Trust-Regulator Framework for Learning under Unobservable Feedback Reliability
China Mobile Research Institute（中国移动研究院）；China Mobile GBA (Greater Bay Area) Innovation Institute（中国移动粤港澳大湾区创新研究院）；Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）
