In the last decade, Reinforcement Learning (RL) has achieved remarkable success in the control and decision-making of complex dynamical systems. However, most RL algorithms rely on the Markov Decision Process assumption, which is violated in practical cyber-physical systems affected by sensing delays, actuation latencies, and communication constraints. Such time delays introduce memory effects that can significantly degrade performance and compromise stability, particularly in networked and multi-agent environments. This paper presents a comprehensive survey of RL methods designed to address time delays in control systems. We first formalize the main classes of delays and analyze their impact on the Markov property. We then systematically categorize existing approaches into five major families: state augmentation and history-based representations, recurrent policies with learned memory, predictor-based and model-aware methods, robust and domain-randomized training strategies, and safe RL frameworks with explicit constraint handling. For each family, we discuss underlying principles, practical advantages, and inherent limitations. A comparative analysis highlights key trade-offs among these approaches and provides practical guidelines for selecting suitable methods under different delay characteristics and safety requirements. Finally, we identify open challenges and promising research directions, including stability certification, large-delay learning, multi-agent communication co-design, and standardized benchmarking. This survey aims to serve as a unified reference for researchers and practitioners developing reliable RL-based controllers in delay-affected cyber-physical systems.
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；Intelligent Game and Decision Lab (IGDL)（智能游戏与决策实验室）；Institute of Military Transportation（军事交通运输研究院）；School of Artificial Intelligence, Hebei University of Technology（河北工业大学人工智能学院）
Learning Personalized Ad Impact via Contextual Reinforcement Learning under Delayed Rewards
Department of Statistics University of Chicago（芝加哥大学统计学系）；Mendoza College of Business University of Notre Dame（诺特丹大学梅多扎商学院）；Department of Computer Science and Data Science University of Chicago（芝加哥大学计算机科学与数据科学系）
Learning Decentralized Routing Policies via Graph Attention-based Multi-Agent Reinforcement Learning in Lunar Delay-Tolerant Networks
Telecommunications Research Institute, University of Málaga（马德里大学电信研究所）；Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）；Department of Mechanical Engineering, Johns Hopkins University（约翰霍普金斯大学机械工程系）
Reinforcement Learning via Conservative Agent for Environments with Random Delays
Department of Convergence IT Engineering, Pohang University of Science and Technology（融合信息技术工程系，POSTECH）；Department of Electrical Engineering, Pohang University of Science and Technology（电气工程系，POSTECH）
Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays
The University of Liverpool（利物浦大学）；Northwestern University（西北大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；National Taiwan University（国立台湾大学）；Nanyang Technological University（南洋理工大学）；The Swiss AI Lab IDSIA/USI/SUPSI（瑞士IDSIA人工智能实验室（隶属于瑞士意大利语区大学与瑞士应用科学大学））；The University of Southampton（南安普顿大学）
Deep Reinforcement Learning for Real-Time Ground Delay Program Revision and Corresponding Flight Delay Assignments
University of California at Berkeley（加州大学伯克利分校）；Northwestern University（西北大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；University of British Columbia（不列颠哥伦比亚大学）；University of Toronto（多伦多大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）
Deep Reinforcement Learning for Delay-Oriented IoT Task Scheduling in Space-Air-Ground Integrated Network
University of Waterloo（滑铁卢大学）；Zhejiang University（浙江大学）；Huazhong University of Science and Technology（华中科技大学）；Central South University（中南大学）；Xidian University（西安电子科技大学）
Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling
College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）；Intelligent Game and Decision Lab (IGDL)（智能游戏与决策实验室）；Institute of Military Transportation（军事交通运输研究院）；School of Artificial Intelligence, Hebei University of Technology（人工智能学院，河北工业大学）
Handling Delay in Real-Time Reinforcement Learning
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；University of Calgary（卡尔加里大学）；IBM Research（IBM研究院）；University of Cambridge（剑桥大学）；CIFAR AI Chair（加拿大高级研究院人工智能主席项目）
Directly Forecasting Belief for Reinforcement Learning with Delays
University of Southampton（南安普顿大学）；Northwestern University（西北大学）；National Taiwan University（台湾大学）；The Swiss AI Lab IDSIA/USI/SUPSI（瑞士人工智能实验室 IDSIA/USI/SUPSI）
Model Mediated Teleoperation with a Hand-Arm Exoskeleton in Long Time Delays Using Reinforcement Learning
German Aerospace Center（德国航空航天中心）；Hamburg University（汉堡大学）；Technical University of Munich（慕尼黑工业大学）
远程机器人系统必须适应新的环境条件，并处理长时间延迟造成的高度不确定性。强化学习（RL）作为人类智能的最佳替代方案之一，可能会提供一个解决方案来应对这些问题。本文提出将RL与模型中介遥操作（MMT）概念相结合。遥控操作员与提供即时反馈的模拟虚拟环境进行交互。虽然来自真实环境的反馈是延迟的，但来自模型的反馈是即时的，这导致了高透明度。MMT与两层智能系统相结合实现。第一层利用动态运动基元（Dynamic Movement Primitives，缩写为MOM），它可以解释化身环境中的某些变化。并且，第二层使用RL方法解决模型中的不确定性所引起的问题。还提供了增强现实来融合化身设备和遥控操作员的虚拟环境模型。在DLR的Exodex Adam手臂触觉外骨骼上实现，结果表明，当演示后对象位置发生变化时，RL方法能够找到不同的解决方案。结果还表明，DMPs在适应新的条件下是有效的，其中没有不确定性。
Multi-Agent Deep Reinforcement Learning for Cost- and Delay-Sensitive Virtual Network Function Placement and Routing
Beijing University of Posts and Telecommunications（北京邮电大学）；SUTD（新加坡科技设计大学）；Commonwealth Scientific and Industrial Research Organisation (CSIRO)（联邦科学与工业研究组织（CSIRO））；NTU（南洋理工大学）
Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications
VTT Technical Research Centre of Finland（芬兰VTT技术研究中心）；University of Electro-Communications（电气通信大学）；Zhejiang University（浙江大学）；University of Oslo（奥斯陆大学）；University of Oulu（奥卢大学）
Revenue and Energy Efficiency-Driven Delay Constrained Computing Task Offloading and Resource Allocation in a Vehicular Edge Computing Network: A Deep Reinforcement Learning Approach
Power Allocation for Delay Optimization in Device-to-Device Networks: A Graph Reinforcement Learning Approach
National Mobile Communications Research Laboratory, Southeast University（东南大学国家移动通信研究中心）；College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）；Department of Electrical and Computer Engineering, University of California, Santa Cruz（加州大学圣克鲁兹分校电子与计算机工程系）
