数字孪生网络的双时间尺度同步与迁移：一种多智能体深度强化学习方法
Two-Timescale Synchronization and Migration for Digital Twin Networks: A Multi-Agent Deep Reinforcement Learning Approach
AI总结：
本文针对移动用户移动性导致的数字孪生同步与迁移难题，提出双时间尺度框架，并设计Beta-HAPPO多智能体深度强化学习方法以最小化长期能耗。
摘要：
数字孪生（DTs）已成为一种有前景的使能技术，用于表示物理世界的实时状态并实现自维持系统。在实践中，物理设备的数字孪生，例如移动用户（MUs）的数字孪生，通常部署在多接入边缘计算（MEC）网络中，以减少时延。为确保数字孪生的准确性和保真度，移动用户必须定期与其数字孪生同步状态。然而，移动用户的移动性给数字孪生同步带来了重大挑战。首先，移动用户的移动性会触发数字孪生迁移，这可能导致同步失败。其次，移动用户需要频繁与其数字孪生同步以确保数字孪生保真度。尽管如此，由移动用户移动性引起的数字孪生在MEC服务器之间的迁移可能并不频繁发生。据此，我们提出了一种考虑可靠性的双时间尺度数字孪生同步与迁移框架，通过建立一个非凸随机问题来最小化移动用户的长期平均能耗。我们利用李雅普诺夫理论将可靠性约束进行转化，并将新问题重新表述为部分可观测马尔可夫决策过程（POMDP）。此外，我们开发了一种基于Beta分布的异构智能体近端策略优化（Beta-HAPPO）方法来求解该问题。数值结果表明，与其他基准方法相比，我们提出的Beta-HAPPO方法在节能方面取得了显著提升。
英文摘要：
Digital twins (DTs) have emerged as a promising enabler for representing the real-time states of physical worlds and realizing self-sustaining systems. In practice, DTs of physical devices, such as mobile users (MUs), are commonly deployed in multi-access edge computing (MEC) networks for the sake of reducing latency. To ensure the accuracy and fidelity of DTs, it is essential for MUs to regularly synchronize their status with their DTs. However, MU mobility introduces significant challenges to DT synchronization. Firstly, MU mobility triggers DT migration which could cause synchronization failures. Secondly, MUs require frequent synchronization with their DTs to ensure DT fidelity. Nonetheless, DT migration among MEC servers, caused by MU mobility, may occur infrequently. Accordingly, we propose a two-timescale DT synchronization and migration framework with reliability consideration by establishing a non-convex stochastic problem to minimize the long-term average energy consumption of MUs. We use Lyapunov theory to convert the reliability constraints and reformulate the new problem as a partially observable Markov decision-making process (POMDP). Furthermore, we develop a heterogeneous agent proximal policy optimization with Beta distribution (Beta-HAPPO) method to solve it. Numerical results show that our proposed Beta-HAPPO method achieves significant improvements in energy savings when compared with other benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
