Pareto确定性策略梯度及其在5G大规模MIMO网络中的应用
Pareto Deterministic Policy Gradients and Its Application in 5G Massive MIMO Networks
摘要：
在本文中，我们考虑通过强化学习（RL）方法联合优化小区负载平衡和网络吞吐量，其中小区间切换（即，用户关联分配）和大规模MIMO天线倾斜被配置为要学习的RL策略。我们使用RL的基本原理是规避分析建模用户移动性和网络动态的挑战。为了实现这种联合优化，我们将向量奖励集成到RL值网络中，并通过单独的策略网络进行RL操作。我们将这种方法称为帕累托确定性策略梯度（PDPG）。它是一种行动者-批评者、无模型和确定性的策略算法，可以处理耦合目标，具有以下两个优点：1）它通过利用向量奖励的自由度而不是选择手工标量奖励来解决优化问题; 2）可以显着减少多个策略的交叉验证。因此，RL使能网络以自组织的方式表现：它通过测量历史了解底层用户移动性，以在没有环境假设的情况下主动操作切换和天线倾斜。我们的数值计算表明，引入RL方法优于基于标量奖励的方法。同时，为了自成一体，包括一个理想的静态优化的暴力搜索求解器作为基准。比较结果表明，RL方法执行以及这种理想的策略，虽然前者是有限的环境观测和较低的动作频率的约束，而后者有充分的访问用户的移动性。我们所介绍的方法的收敛性也测试在不同的用户移动性环境下，我们的测量数据从一个真实的场景。
英文摘要：
In this paper, we consider jointly optimizing cell load balance and network throughput via a reinforcement learning (RL) approach, where inter-cell handover (i.e., user association assignment) and massive MIMO antenna tilting are configured as the RL policy to learn. Our rationale behind using RL is to circumvent the challenges of analytically modeling user mobility and network dynamics. To accomplish this joint optimization, we integrate vector rewards into the RL value network and conduct RL action via a separate policy network. We name this method as Pareto deterministic policy gradients (PDPG). It is an actor-critic, model-free and deterministic policy algorithm which can handle the coupling objectives with the following two merits: 1) It solves the optimization via leveraging the degree of freedom of vector reward as opposed to choosing handcrafted scalar-reward; 2) Cross-validation over multiple policies can be significantly reduced. Accordingly, the RL enabled network behaves in a self-organized way: It learns out the underlying user mobility through measurement history to proactively operate handover and antenna tilt without environment assumptions. Our numerical evaluation demonstrates that the introduced RL method outperforms scalar-reward based approaches. Meanwhile, to be self-contained, an ideal static optimization based brute-force search solver is included as a benchmark. The comparison shows that the RL approach performs as well as this ideal strategy, though the former one is constrained with limited environment observations and lower action frequency, whereas the latter ones have full access to the user mobility. The convergence of our introduced approach is also tested under different user mobility environment based on our measurement data from a real scenario.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
