arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.09236cs.NI
收藏

在线网络资源分配的基于模型的强化学习框架

Model-Based Reinforcement Learning Framework of Online Network Resource Allocation

Bahador Bakhshi, Josep Mangues-Bafalluy

摘要:

用于业务提供的在线网络资源分配(ONRA)是通信网络中的基本问题。作为一种不确定性条件下的序贯决策问题,将强化学习(RL)方法引入ONRA是一种很有前途的方法。但是,RL解决方案受到样本复杂性问题的影响;即,需要与环境进行大量的互动,以找到一个有效的政策。这是一个障碍,利用强化学习的ONRA一方面,这是不切实际的训练的强化学习代理离线,由于缺乏有关未来的请求的信息,另一方面,在线训练在真实的网络导致显着的性能损失,因为在延长的学习时间的次优政策。这种性能下降在非静态ONRA中甚至更高,其中代理应该随着服务请求的变化不断调整策略。为了解决这个问题,我们开发了一个通用的资源分配框架,命名为雷达,使用基于模型的强化学习的一类ONRA问题,每个动作的即时奖励。RADAR通过对环境的真实交互进行训练,建立环境模型,在背景中探索状态空间,在决策时使用合成样本进行决策,从而提高了样本效率。将RADAR应用于多域服务联盟问题,通过选择合适的域进行服务请求部署,实现利润最大化,显示了其持续学习能力,相对于传统的服务联盟算法,性能提高了44%。标准的无模型RL解决方案。

英文摘要:

Online Network Resource Allocation (ONRA) for service provisioning is a fundamental problem in communication networks. As a sequential decision-making under uncertainty problem, it is promising to approach ONRA via Reinforcement Learning (RL). But, RL solutions suffer from the sample complexity issue; i.e., a large number of interactions with the environment needed to find an efficient policy. This is a barrier to utilize RL for ONRA as on one hand, it is not practical to train the RL agent offline due to lack of information about future requests, and on the other hand, online training in the real network leads to significant performance loss because of the sub-optimal policy during the prolonged learning time. This performance degradation is even higher in non-stationary ONRA where the agent should continually adapt the policy with the changes in service requests. To deal with this issue, we develop a general resource allocation framework, named RADAR, using model-based RL for a class of ONRA problems with the known immediate reward of each action. RADAR improves sample efficiency via exploring the state space in the background and exploiting the policy in the decision-time using synthetic samples by the model of the environment, which is trained by real interactions. Applying RADAR on the multi-domain service federation problem, to maximize profit via selecting proper domains for service requests deployment, shows its continual learning capability and up to 44% performance improvement w.r.t. the standard model-free RL solution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑