arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.16142cs.NIcs.LG
收藏

联合O-RAN/MEC编排的深度强化学习贝叶斯框架

A Bayesian Framework of Deep Reinforcement Learning for Joint O-RAN/MEC Orchestration

Centre for Wireless Communications, University of Oulu(奥卢大学无线通信中心)

Fahri Wisnu Murti, Samad Ali, Matti Latva-aho

AI总结:

本文提出一种贝叶斯深度强化学习框架,联合编排O-RAN/MEC以最小化运营成本并最大化性能,采用DDQN与动作分支处理大规模状态和离散动作,并通过汤普森采样加速收敛,仿真显示奖励提升32%。

摘要:

多接入边缘计算(MEC)可以在商品化平台上与开放无线接入网(O-RAN)一起实现,以提供低成本的部署并将服务更接近最终用户。本文提出了一种基于贝叶斯深度强化学习(RL)框架的联合O-RAN/MEC编排方法,该方法联合控制O-RAN功能拆分、跨地理分布平台的O-RAN/MEC服务所分配的资源及托管位置,以及每个O-RAN/MEC数据流的路由。其目标是在适应可能随时间变化的O-RAN/MEC需求和资源可用性的同时,最小化长期整体网络运营成本并最大化MEC性能指标。该编排问题被建模为马尔可夫决策过程(MDP)。然而,系统由多个基站(BS)组成,它们共享相同的资源并服务于异构需求,其参数之间存在非平凡的关系。因此,找到底层系统的精确模型是不切实际的,所建立的MDP具有大规模状态空间和多维离散动作。为了解决这种建模和维度问题,我们为解决方案框架提出了一种新颖的无模型RL代理。该代理基于Double Deep Q-network(DDQN)构建,以处理大规模状态空间,并结合了动作分支(一种动作分解方法),以线性增加的复杂度有效处理多维离散动作。此外,提出了一种基于汤普森采样的贝叶斯框架下的高效探索-利用策略,以提高学习性能并加速收敛。使用符合O-RAN的模型进行了基于轨迹的仿真。结果表明,我们的方法具有数据效率(即收敛更快),并且返回的奖励比其非贝叶斯版本高出32%。

英文摘要:

Multi-access Edge Computing (MEC) can be implemented together with Open Radio Access Network (O-RAN) over commodity platforms to offer low-cost deployment and bring the services closer to end-users. In this paper, a joint O-RAN/MEC orchestration using a Bayesian deep reinforcement learning (RL)-based framework is proposed that jointly controls the O-RAN functional splits, the allocated resources and hosting locations of the O-RAN/MEC services across geo-distributed platforms, and the routing for each O-RAN/MEC data flow. The goal is to minimize the long-term overall network operation cost and maximize the MEC performance criterion while adapting possibly time-varying O-RAN/MEC demands and resource availability. This orchestration problem is formulated as Markov decision process (MDP). However, the system consists of multiple BSs that share the same resources and serve heterogeneous demands, where their parameters have non-trivial relations. Consequently, finding the exact model of the underlying system is impractical, and the formulated MDP renders in a large state space with multi-dimensional discrete action. To address such modeling and dimensionality issues, a novel model-free RL agent is proposed for our solution framework. The agent is built from Double Deep Q-network (DDQN) that tackles the large state space and is then incorporated with action branching, an action decomposition method that effectively addresses the multi-dimensional discrete action with linear increase complexity. Further, an efficient exploration-exploitation strategy under a Bayesian framework using Thomson sampling is proposed to improve the learning performance and expedite its convergence. Trace-driven simulations are performed using an O-RAN-compliant model. The results show that our approach is data-efficient (i.e., converges faster) and increases the returned reward by 32\% than its non-Bayesian version.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑