This paper analyzes the simulation to reality gap in reinforcement learning (RL) cyber-physical systems with fractional delays (i.e. delays that are non-integer multiple of the sampling period). The consideration of fractional delay has important implications on the nature of the cyber-physical system considered. Systems with delays are non-Markovian, and the system state vector needs to be extended to make the system Markovian. We show that this is not possible when the delay is in the output, and the problem would always be non-Markovian. Based on this analysis, a sampling scheme is proposed that results in efficient RL training and agents that perform well in realistic multirotor unmanned aerial vehicle simulations. We demonstrate that the resultant agents do not produce excessive oscillations, which is not the case with RL agents that do not consider time delay in the model.
Enhancing Cyber-Resilience in Integrated Energy System Scheduling with Demand Response Using Deep Reinforcement Learning
School of Electrical Engineering, Northeast Electric Power University（东北电力大学电气工程学院）；State Grid XiXian New Area Power Supply Company（国家电网西安新区供电公司）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；The Hong Kong University of Science and Technology（香港科技大学）；Institute of Energy Technology, Aalborg University（奥尔堡大学能源技术研究所）；Illinois Institute of Technology（伊利诺伊理工大学）
Model-Free Reinforcement Learning Control for Resilient Cyber-Physical Systems
Departmento de Ingenier\'ia El\'ectrica, Universidad de Concepci\'on, Concepci\'on, Chile (e-mail: )；Department of Electrical \& Computer Engineering, University of Alberta, Edmonton, T6G 1H9, Alberta, AB, Canada (e-mail: )；Department of Chemical；Biological Engineering, University of British Columbia, Vancouver, BC V6T 1Z3, Canada ( )；Department of Chemical \& Materials Engineering, University of Alberta, Edmonton, T6G 1H9, Alberta, AB, Canada (e-mail: )
Communication-aware Wide-Area Damping Control using Risk-Constrained Reinforcement Learning
Optimization and Control Group, Pacific Northwest National Laboratory（优化与控制组，太平洋西北国家实验室）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（人工智能与自动化学院，华中科技大学）；Elmore School of Electrical and Computer Engineering, Purdue University（Elmore电气与计算机工程学院，普渡大学）；Chandra Family Department of Electrical & Computer Engineering, The University of Texas at Austin（Chandra家族电气与计算机工程系，德克萨斯大学奥斯汀分校）
Game-Theoretic Resilience Framework for Cyber-Physical Microgrids using Multi-Agent Reinforcement Learning
ABB Ability Innovation Center, Hyderabad（ABB Ability创新中心，海得拉巴）；Indian Institute of Technology Tirupati（印度理工学院特里普蒂学院）；Department of Electrical Engineering, Indian Institute of Technology (ISM), Dhanbad（印度理工学院（ISM）电子工程系，丹巴德）；Indian Institute of Technology (ISM), Dhanbad（印度理工学院（ISM），丹巴德）
Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays
The University of Liverpool（利物浦大学）；Northwestern University（西北大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；National Taiwan University（国立台湾大学）；Nanyang Technological University（南洋理工大学）；The Swiss AI Lab IDSIA/USI/SUPSI（瑞士IDSIA人工智能实验室（隶属于瑞士意大利语区大学与瑞士应用科学大学））；The University of Southampton（南安普顿大学）
Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay
College of Information Science and Technology, Donghua University（东华大学信息科学与技术学院）；College of Information Engineering, Shanghai Maritime University（上海海事大学信息工程学院）
