arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.07967eess.SYcs.LGcs.MAcs.SY
收藏

HetNet中分布式训练执行多Agent强化学习的功率控制

Distributed-Training-and-Execution Multi-Agent Reinforcement Learning for Power Control in HetNet

Imperial College London(帝国理工学院)

Kaidi Xu, Nguyen Van Huynh, Geoffrey Ye Li

摘要:

在异构网络(HetNet)中,小小区和宏小区的重叠导致严重的跨层干扰。虽然已有一些方法来解决这个问题,但它们通常需要全局信道状态信息,而这在实际中是很难获得的,并且得到的次优功率分配策略具有很高的计算复杂度。为了克服这些限制,我们提出了一种基于多代理深度强化学习(MADRL)的HetNet功率控制方案,其中每个接入点根据本地信息独立做出功率控制决策。为了促进代理之间的合作,我们开发了一个基于惩罚的Q学习(PQL)算法的MADRL系统。通过在损失函数中引入正则化项,每个代理在重访状态时倾向于选择具有高回报的经验动作,从而减慢了策略更新速度。通过这种方式,代理的策略可以更容易地被其他代理学习,从而导致更有效的协作过程。然后,我们在考虑HetNet实现建议的PQL,并将其与其他分布式训练和执行(DTE)算法进行比较。仿真结果表明,我们提出的PQL可以学习所需的功率控制策略,从一个动态的环境中,用户的位置发生变化,并优于现有的DTE MADRL算法。

英文摘要:

In heterogeneous networks (HetNets), the overlap of small cells and the macro cell causes severe cross-tier interference. Although there exist some approaches to address this problem, they usually require global channel state information, which is hard to obtain in practice, and get the sub-optimal power allocation policy with high computational complexity. To overcome these limitations, we propose a multi-agent deep reinforcement learning (MADRL) based power control scheme for the HetNet, where each access point makes power control decisions independently based on local information. To promote cooperation among agents, we develop a penalty-based Q learning (PQL) algorithm for MADRL systems. By introducing regularization terms in the loss function, each agent tends to choose an experienced action with high reward when revisiting a state, and thus the policy updating speed slows down. In this way, an agent's policy can be learned by other agents more easily, resulting in a more efficient collaboration process. We then implement the proposed PQL in the considered HetNet and compare it with other distributed-training-and-execution (DTE) algorithms. Simulation results show that our proposed PQL can learn the desired power control policy from a dynamic environment where the locations of users change episodically and outperform existing DTE MADRL algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑