For a multi-cell, multi-user, cellular network downlink sum-rate maximization through power allocation is a nonconvex and NP-hard optimization problem. In this paper, we present an effective approach to solving this problem through single- and multi-agent actor-critic deep reinforcement learning (DRL). Specifically, we use finite-horizon trust region optimization. Through extensive simulations, we show that we can simultaneously achieve higher spectral efficiency than state-of-the-art optimization algorithms like weighted minimum mean-squared error (WMMSE) and fractional programming (FP), while offering execution times more than two orders of magnitude faster than these approaches. Additionally, the proposed trust region methods demonstrate superior performance and convergence properties than the Advantage Actor-Critic (A2C) DRL algorithm. In contrast to prior approaches, the proposed decentralized DRL approaches allow for distributed optimization with limited CSI and controllable information exchange between BSs while offering competitive performance and reduced training times.
In this work, we propose a convolutional neural network (CNN) based low-complexity approach for downlink (DL) channel estimation (CE) in frequency division duplex (FDD) systems. In contrast to existing work, we use training data which solely stems from the uplink (UL) domain. This allows to learn the CNN centralized at the base station (BS). After training, the network parameters are offloaded to mobile terminals (MTs) within the coverage area of the BS. The MTs can then obtain channel state information (CSI) of the MIMO channels with the low-complexity CNN estimator. This circumvents the nece…
In this work, we propose a globally optimal joint successive interference cancellation (SIC) ordering and power allocation (JSPA) algorithm for the sum-rate maximization problem in downlink multi-cell non-orthogonal multiple access (NOMA) systems. The proposed algorithm is based on the exploration of base stations (BSs) power consumption, and closed-form of optimal powers obtained for each cell. Although the optimal JSPA algorithm scales well with larger number of users, it is still exponential in the number of cells. For any suboptimal decoding order, we propose a low-complexity near-optimal …
This paper compares centralized and distributed methods to solve the power minimization problem with quality-of-service (QoS) constraints in the downlink (DL) of multi-cell Massive multiple-input multiple-output (MIMO) systems. In particular, we study the computational complexity, number of parameters that need to be exchanged between base stations (BSs), and the convergence of iterative implementations. Although a distributed implementation based on dual decomposition (which only requires statistical channel knowledge at each BS) typically converges to the global optimum after a few iteration…
We study the problem of associating mobile terminals to base stations in a linear interference network, with the goal of maximizing the average rate achieved over both the uplink and downlink sessions. The cell association decision is made at a centralized cloud level, with access to global network topology information. More specifically, given the constraint that each mobile terminal can be associated to a maximum of N base stations at once, we characterize the maximum achievable pre-log factor (degrees of freedom) and the corresponding cell association pattern. Interestingly, the result indi…
Qatar University（卡塔尔大学）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）；Politecnico di Torino（都灵理工大学）
In this work, we examine an intelligent reflecting surface (IRS) assisted downlink non-orthogonal multiple access (NOMA) scenario with the aim of maximizing the sum rate of users. The optimization problem at the IRS is quite complicated, and non-convex, since it requires the tuning of the phase shift reflection matrix. Driven by the rising deployment of deep reinforcement learning (DRL) techniques that are capable of coping with solving non-convex optimization problems, we employ DRL to predict and optimally tune the IRS phase shift matrices. Simulation results reveal that IRS assisted NOMA ba…
Base station (BS) cooperation can turn unwanted interference to useful signal energy for enhancing system performance. In the cooperative downlink, zero-forcing beamforming (ZFBF) with a simple scheduler is well known to obtain nearly the performance of the capacity-achieving dirty-paper coding. However, the centralized ZFBF approach is prohibitively complex as the network size grows. In this paper, we devise message passing algorithms for realizing the regularized ZFBF (RZFBF) in a distributed manner using belief propagation. In the proposed methods, the overall computational cost is decompos…
In this paper, a novel framework is proposed to optimize the downlink multi-user communication of a millimeter wave base station, which is assisted by a reconfigurable intelligent reflector (IR). In particular, a channel estimation approach is developed to measure the channel state information (CSI) in real-time. First, for a perfect CSI scenario, the precoding transmission of the BS and the reflection coefficient of the IR are jointly optimized, via an iterative approach, so as to maximize the sum of downlink rates towards multiple users. Next, in the imperfect CSI scenario, a distributional …
With the high development of wireless communication techniques, it is widely used in various fields for convenient and efficient data transmission. Different from commonly used assumption of the time-invariant wireless channel, we focus on the research on the time-varying wireless downlink channel to get close to the practical situation. Our objective is to gain the maximum value of sum rate in the time-varying channel under the some constraints about cut-off signal-to-interference and noise ratio (SINR), transmitted power and beamforming. In order to adapt the rapid changing channel, we aband…
In this paper, we consider a multi-cell network where every base station (BS) serves multiple users with an antenna array. Each user is associated with only one BS and has a single antenna. Assume that only long-term channel state information (CSI) is available in the system. The objective is to minimize the network downlink transmission power needed to meet the users' signal-to-interference-plus-noise ratio (SINR) requirements. For this objective, we propose an asynchronous distributed beamforming and power control algorithm which provides the same optimal solution as given by centralized alg…
In this paper, a novel communication framework that uses an unmanned aerial vehicle (UAV)-carried intelligent reflector (IR) is proposed to enhance multi-user downlink transmissions over millimeter wave (mmWave) frequencies. In order to maximize the downlink sum-rate, the optimal precoding matrix (at the base station) and reflection coefficient (at the IR) are jointly derived. Next, to address the uncertainty of mmWave channels and maintain line-of-sight links in a real-time manner, a distributional reinforcement learning approach, based on quantile regression optimization, is proposed to lear…
This paper considers a multicloud radio access network (M-CRAN), wherein each cloud serves a cluster of base-stations (BS's) which are connected to the clouds through high capacity digital links. The network comprises several remote users, where each user can be connected to one (and only one) cloud. This paper studies the user-to-cloud-assignment problem by maximizing a network-wide utility subject to practical cloud connectivity constraints. The paper solves the problem by using an auction-based iterative algorithm, which can be implemented in a distributed fashion through a reasonable excha…
This paper studies multi-agent deep reinforcement learning (MADRL) based resource allocation methods for multi-cell wireless powered communication networks (WPCNs) where multiple hybrid access points (H-APs) wirelessly charge energy-limited users to collect data from them. We design a distributed reinforcement learning strategy where H-APs individually determine time and power allocation variables. Unlike traditional centralized optimization algorithms which require global information collected at a central unit, the proposed MADRL technique models an H-AP as an agent producing its action base…
Massive multi-user multiple-input multiple-output (MU-MIMO) enables significant gains in spectral efficiency and link reliability compared to conventional small-scale MIMO technology. Furthermore, linear precoders, e.g., using zero forcing or Wiener filter (WF) precoding, are sufficient to achieve excellent error-rate performance in the massive MU-MIMO downlink. However, these methods necessitate centralized processing at the base-station (BS), which causes (i) excessively high interconnect and chip input/output data rates, and (ii) high implementation complexity. We propose two feedforward ar…
This paper addresses coordinated downlink beamforming optimization in multicell time-division duplex (TDD) systems where a small number of parameters are exchanged between cells but with no data sharing. With the goal to reach the point on the Pareto boundary with max-min rate fairness, we first develop a two-step centralized optimization algorithm to design the joint beamforming vectors. This algorithm can achieve a further sum-rate improvement over the max-min optimal performance, and is shown to guarantee max-min Pareto optimality for scenarios with two base stations (BSs) each serving a si…
Coordinated information processing by the base stations of multi-cell wireless networks enhances the overall quality of communication in the network. Such coordinations for optimizing any desired network-wide quality of service (QoS) necessitate the base stations to acquire and share some channel state information (CSI). With perfect knowledge of channel states, the base stations can adjust their transmissions for achieving a network-wise QoS optimality. In practice, however, the CSI can be obtained only imperfectly. As a result, due to the uncertainties involved, the network is not guaranteed…
Deep Reinforcement Learning-Based Precoding for Multi-RIS-Aided Multiuser Downlink Systems with Practical Phase Shift
Research Center for Information Technology Innovation (CITI), Academia Sinica（信息科技创新研究中心（CITI），学术院）；Institute of Communications Engineering, National Sun Yat-sen University（通讯工程研究所，国立中山大学）；Bradley Department of Electrical and Computer Engineering, Virginia Tech（电气与计算机工程系，弗吉尼亚理工大学）
In this paper, we investigate the uplink transmit power optimization problem in cell-free (CF) extremely large-scale multiple-input multiple-output (XL-MIMO) systems. Instead of applying the traditional methods, we propose two signal processing architectures: the centralized training and centralized execution with fuzzy logic as well as the centralized training and decentralized execution with fuzzy logic, respectively, which adopt the amalgamation of multi-agent reinforcement learning (MARL) and fuzzy logic to solve the design problem of power control for the maximization of the system spectr…
Intelligent reflecting surface (IRS) is a new promising technology that is able to reconfigure the wireless propagation channel via smart and passive signal reflection. In this paper, we investigate the capacity region of a two-user communication network with one access point (AP) aided by $M$ IRS elements for enhancing the user-AP channels, where the IRS incurs negligible delay, thus the user-AP channels via the IRS follow the classic discrete memoryless channel model. In particular, we consider two practical IRS deployment strategies that lead to different effective channels between the user…
This paper considers a cell-free massive multiple-input multiple-output (MIMO) system that consists of a large number of geographically distributed access points (APs) serving multiple users via coherent joint transmission. The downlink performance of the system is evaluated, with maximum ratio and regularized zero-forcing precoding, under two optimization objectives for power allocation: sum spectral efficiency (SE) maximization and proportional fairness. We present iterative centralized algorithms for solving these problems. Aiming at a less computationally complex and also distributed scala…
Cell-free Massive MIMO systems consist of a large number of geographically distributed access points (APs) that serve users by coherent joint transmission. Downlink power allocation is important in these systems, to determine which APs should transmit to which users and with what power. If the system is implemented correctly, it can deliver a more uniform user performance than conventional cellular networks. To this end, previous works have shown how to perform system-wide max-min fairness power allocation when using maximum ratio precoding. In this paper, we first generalize this method to ar…
Huawei Technologies German Research Center（华为技术德国研究中心）
Radio resources in vehicle-to-vehicle (V2V) communication can be scheduled either by a centralized scheduler residing in the network (e.g., a base station in case of cellular systems) or a distributed scheduler, where the resources are autonomously selected by the vehicles. The former approach yields a considerably higher resource utilization in case the network coverage is uninterrupted. However, in case of intermittent or out-of-coverage, due to not having input from centralized scheduler, vehicles need to revert to distributed scheduling. Motivated by recent advances in reinforcement learni…
This paper investigates the energy efficiency (EE) optimization in downlink multi-cell massive multiple-input multiple-output (MIMO). In our research, the statistical channel state information (CSI) is exploited to reduce the signaling overhead. To maximize the minimum EE among the neighbouring cells, we design the transmit covariance matrices for each base station (BS). Specifically, optimization schemes for this max-min EE problem are developed, in the centralized and distributed ways, respectively. To obtain the transmit covariance matrices, we first find out the closed-form optimal transmi…
We consider an Intelligent Reflecting Surface (IRS)-aided multiple-input single-output (MISO) system for downlink transmission. We compare the performance of Deep Reinforcement Learning (DRL) and conventional optimization methods in finding optimal phase shifts of the IRS elements to maximize the user signal-to-noise (SNR) ratio. Furthermore, we evaluate the robustness of these methods to channel impairments and changes in the system. We demonstrate numerically that DRL solutions show more robustness to noisy channels and user mobility.
In this paper, the downlink packet scheduling problem for cellular networks is modeled, which jointly optimizes throughput, fairness and packet drop rate. Two genie-aided heuristic search methods are employed to explore the solution space. A deep reinforcement learning (DRL) framework with A2C algorithm is proposed for the optimization problem. Several methods have been utilized in the framework to improve the sampling and training efficiency and to adapt the algorithm to a specific scheduling problem. Numerical results show that DRL outperforms the baseline algorithm and achieves similar perf…
The Cooperative Rate-Splitting (CRS) scheme, proposed evolves from conventional Rate Splitting (RS) and relies on forwarding a portion of the RS message by the relaying users. In terms of secrecy enhancement, it has been shown that CRS outperforms its non-cooperative counterpart for a two-user Multiple Input Single Output (MISO) Broadcast Channel (BC). Given the massive connectivity requirement of 6G, we have generalized the existing secure two-user CRS framework to the multi-user framework, where the highest-security users must be selected as the relay nodes. This paper addresses the problem …
CentraleSupelec（巴黎中央高等电力学院）；Université Paris-Saclay（巴黎萨克雷大学）；University of Cassino and Southern Lazio（卡西诺和南拉齐奥大学）；Mathematical and Algorithmic Sciences Lab, Huawei France R&D（华为法国研发数学与算法科学实验室）
In this paper, we develop a multi-agent reinforcement learning (MARL) framework to obtain online power control policies for a large energy harvesting (EH) multiple access channel, when only causal information about the EH process and wireless channel is available. In the proposed framework, we model the online power control problem as a discrete-time mean-field game (MFG), and analytically show that the MFG has a unique stationary solution. Next, we leverage the fictitious play property of the mean-field games, and the deep reinforcement learning technique to learn the stationary solution of t…
Optimizing radio transmission power and user data rates in wireless systems via power control requires an accurate and instantaneous knowledge of the system model. While this problem has been extensively studied in the literature, an efficient solution approaching optimality with the limited information available in practical systems is still lacking. This paper presents a reinforcement learning framework for power control and rate adaptation in the downlink of a radio access network that closes this gap. We present a comprehensive design of the learning framework that includes the characteriz…
The basic idea of device-to-device (D2D) communication is that pairs of suitably selected wireless devices reuse the cellular spectrum to establish direct communication links, provided that the adverse effects of D2D communication on cellular users is minimized and cellular users are given a higher priority in using limited wireless resources. Despite its great potential in terms of coverage and capacity performance, implementing this new concept poses some challenges, in particular with respect to radio resource management. The main challenges arise from a strong need for distributed D2D solu…
K. N. Toosi University of Technology（K. N. 图西理工大学）
This letter investigates an intelligent reflecting surfaces (IRS)-enhanced network from spectral efficiency enhancement point of view for downlink multi-user (MU) multi-input-single-output systems (MISO). In contrast to previous works which mainly focused on alternative optimization methods, we investigate the non-convex joint optimization problem of the active transmit beamforming matrix at the base station together with the passive phase shift matrix at the IRS by utilizing two deep reinforcement learning frameworks, i. e., deep deterministic policy gradient (DDPG) and twin delayed DDPG (TD3…
