数字孪生辅助的深度强化学习用于切片网络中的在线准入控制
Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network
AI总结:
针对切片网络准入控制中DRL模型初始不稳定、收敛慢的问题,提出数字孪生辅助的DRL方案,通过DT预训练辅助加速DRL收敛,相比现有对决深度Q学习模型资源利用率提升超40%,同时保留长期奖励优化能力。
摘要:
5G及未来网络中多样化无线服务的普及推动了网络切片技术的出现。其中,准入控制通过选择性接受服务请求,在实现面向服务的优化目标方面发挥着关键作用。尽管深度强化学习(DRL)凭借其有效性和灵活性成为许多准入控制方法的基础,但DRL模型初始阶段的不稳定性以及过长的收敛延迟阻碍了其在实际网络中的部署。我们提出一种数字孪生(DT)加速的DRL方案来解决这一问题。具体而言,我们首先将准入决策过程建模为半马尔可夫决策过程,随后将其简化为等效的离散时间马尔可夫决策过程,以便于DRL方法的实现。我们构建了一个基于神经网络的DT,其配备了针对排队系统定制的输出层,通过监督学习进行训练,随后用于辅助DRL模型的训练阶段。大量仿真结果表明,与直接训练的最先进的对决深度Q学习模型相比,DT加速的DRL将资源利用率提升了40%以上。在实现这一提升的同时,该方法还保留了模型优化准入过程长期奖励的能力。
英文摘要:
The proliferation of diverse wireless services in 5G and beyond has led to the emergence of network slicing technologies. Among these, admission control plays a crucial role in achieving service-oriented optimization goals through the selective acceptance of service requests. Although deep reinforcement learning (DRL) forms the foundation in many admission control approaches thanks to its effectiveness and flexibility, initial instability with excessive convergence delay of DRL models hinders their deployment in real-world networks. We propose a digital twin (DT) accelerated DRL solution to address this issue. Specifically, we first formulate the admission decision-making process as a semi-Markov decision process, which is subsequently simplified into an equivalent discrete-time Markov decision process to facilitate the implementation of DRL methods. A neural network-based DT is established with a customized output layer for queuing systems, trained through supervised learning, and then employed to assist the training phase of the DRL model. Extensive simulations show that the DT-accelerated DRL improves resource utilization by over 40% compared to the directly trained state-of-the-art dueling deep Q-learning model. This improvement is achieved while preserving the model's capability to optimize the long-term rewards of the admission process.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。