arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.13093cs.NI
收藏

基于平均奖励强化学习的多供应商NFV网络服务委托

Multi-Provider NFV Network Service Delegation via Average Reward Reinforcement Learning

Bahador Bakhshi, Josep Mangues-Bafalluy, Jorge Baranda

摘要:

在多提供商5G/6 G网络中,服务委托使管理域能够联合提供NFV网络服务。接纳控制是在没有服务请求统计分布先验知识的情况下选择合适的部署域以最大化平均利润的基础。本文从多方面分析了一种通用的服务委托联邦契约模型。首先,在已知系统动态的假设下,我们通过将接纳控制问题表示为无限时域马尔可夫决策过程(MDP)并通过动态规划求解来获得理论上的最优性能界。其次,我们应用强化学习来实际解决到达率和离开率未知的问题。由于Q学习最大化折扣奖励,我们证明它不是一个有效的解决方案,由于其对折扣因子的敏感性。然后,我们提出了平均奖励强化学习方法(R-Learning)来找到直接最大化平均利润的策略。最后,我们通过广泛的模拟和使用5Growth平台的实验来评估不同的解决方案。结果证实,所提出的R-Learning解决方案始终优于Q-Learning和贪婪策略。此外,虽然在理想的模拟环境中最多有9%的最优性差距,但它在实验评估中与MDP解决方案竞争。

英文摘要:

In multi-provider 5G/6G networks, service delegation enables administrative domains to federate in provisioning NFV network services. Admission control is fundamental in selecting the appropriate deployment domain to maximize average profit without prior knowledge of service requests' statistical distributions. This paper analyzes a general federation contract model for service delegation in various ways. First, under the assumption of known system dynamics, we obtain the theoretically optimal performance bound by formulating the admission control problem as an infinite-horizon Markov decision process (MDP) and solving it through dynamic programming. Second, we apply reinforcement learning to practically tackle the problem when the arrival and departure rates are not known. As Q-learning maximizes the discounted rewards, we prove it is not an efficient solution due to its sensitivity to the discount factor. Then, we propose the average reward reinforcement learning approach (R-Learning) to find the policy that directly maximizes the average profit. Finally, we evaluate different solutions through extensive simulations and experimentally using the 5Growth platform. Results confirm that the proposed R-Learning solution always outperforms Q-Learning and the greedy policies. Furthermore, while there is at most 9% optimality gap in the ideal simulation environment, it competes with the MDP solution in the experimental assessment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑