We consider a source that wishes to communicate with a destination at a desired rate, over a mmWave network where links are subject to blockage and nodes to failure (e.g., in a hostile military environment). To achieve resilience to link and node failures, we here explore a state-of-the-art Soft Actor-Critic (SAC) deep reinforcement learning algorithm, that adapts the information flow through the network, without using knowledge of the link capacities or network topology. Numerical evaluations show that our algorithm can achieve the desired rate even in dynamic environments and it is robust against blockage.
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Minnesota（明尼苏达大学）
This paper aims to develop resilient transmission mechanisms to suitably distribute traffic across multiple paths in an arbitrary millimeter-wave (mmWave) network. The main contributions include: (a) the development of proactive transmission mechanisms that build resilience against network disruptions in advance, while achieving a high end-to-end packet rate; (b) the design of a heuristic path selection algorithm that efficiently selects (in polynomial time in the network size) multiple proactively resilient paths with high packet rates; and (c) the development of a hybrid scheduling algorithm…
University of Utah（犹他大学）；Idaho National Laboratory（爱达荷国家实验室）
We consider the problem of distributed downlink beam scheduling and power allocation for millimeter-Wave (mmWave) cellular networks where multiple base stations (BSs) belonging to different service operators share the same unlicensed spectrum with no central coordination or cooperation among them. Our goal is to design efficient distributed beam scheduling and power allocation algorithms such that the network-level payoff, defined as the weighted sum of the total throughput and a power penalization term, can be maximized. To this end, we propose a distributed scheduling approach to power alloc…
We study the problem of real-time scheduling in a multi-hop millimeter-wave (mmWave) mesh. We develop a model-free deep reinforcement learning algorithm called Adaptive Activator RL (AARL), which determines the subset of mmWave links that should be activated during each time slot and the power level for each link. The most important property of AARL is its ability to make scheduling decisions within the strict time slot constraints of typical 5G mmWave networks. AARL can handle a variety of network topologies, network loads, and interference models, it can also adapt to different workloads. We…
We investigate the applicability of deep reinforcement learning algorithms to the adaptive initial access beam alignment problem for mmWave communications using the state-of-the-art proximal policy optimization algorithm as an example. In comparison to recent unsupervised learning based approaches developed to tackle this problem, deep reinforcement learning has the potential to address a new and wider range of applications, since, in principle, no (differentiable) model of the channel and/or the whole system is required for training, and only agent-environment interactions are necessary to le…
Directional beamforming is a crucial component for realizing robust wireless communication systems using millimeter wave (mmWave) technology. Beam alignment using brute-force search of the space introduces time overhead while location aided blind beam alignment adds additional hardware requirements to the system. In this paper, we introduce a method for blind beam alignment based on the RF fingerprints of user equipment obtained by the base stations. The proposed system performs blind beam alignment on a multiple base station cellular environment with multiple mobile users using deep reinforce…
Zhejiang University（浙江大学）；University of California, Davis（加州大学戴维斯分校）
The millimeter wave (mmWave) multiuser multiple-input multiple-output (MU-MIMO) systems with discrete lens arrays (DLA) have received great attention due to their simple hardware implementation and excellent performance. In this work, we investigate the joint design of beam selection and digital precoding matrices for mmWave MU-MIMO systems with DLA to maximize the sum-rate subject to the transmit power constraint and the constraints of the selection matrix structure. The investigated non-convex problem with discrete variables and coupled constraints is challenging to solve and an efficient fr…
In this letter, we investigate the hybrid beamforming based on deep reinforcement learning (DRL) for millimeter Wave (mmWave) multi-user (MU) multiple-input-single-output (MISO) system. A multi-agent DRL method is proposed to solve the exploration efficiency problem in DRL. In the proposed method, prioritized replay buffer and more informative reward are applied to accelerate the convergence. Simulation results show that the proposed architecture achieves higher spectral efficiency and less time consumption than the benchmarks, thus is more suitable for practical applications.
In this paper, the downlink packet scheduling problem for cellular networks is modeled, which jointly optimizes throughput, fairness and packet drop rate. Two genie-aided heuristic search methods are employed to explore the solution space. A deep reinforcement learning (DRL) framework with A2C algorithm is proposed for the optimization problem. Several methods have been utilized in the framework to improve the sampling and training efficiency and to adapt the algorithm to a specific scheduling problem. Numerical results show that DRL outperforms the baseline algorithm and achieves similar perf…
In intelligent transportation systems (ITS), vehicles are expected to feature with advanced applications and services which demand ultra-high data rates and low-latency communications. For that, the millimeter wave (mmWave) communication has been emerging as a very promising solution. However, incorporating the mmWave into ITS is particularly challenging due to the high mobility of vehicles and the inherent sensitivity of mmWave beams to dynamic blockages. This article addresses these problems by developing an optimal beam association framework for mmWave vehicular networks under high mobility…
Integrating artificial intelligence (AI) into wireless networks has drawn significant interest in both industry and academia. A common solution is to replace partial or even all modules in the conventional systems, which is often lack of efficiency and robustness due to their ignoring of expert knowledge. In this paper, we take deep reinforcement learning (DRL) based scheduling as an example to investigate how expert knowledge can help with AI module in cellular networks. A simulation platform, which has considered link adaption, feedback and other practical mechanisms, is developed to facilit…
Millimeter Wave (MmWave) communication is one of the key technology of the fifth generation (5G) wireless systems to achieve the expected 1000x data rate. With large bandwidth at mmWave band, the link capacity between users and base stations (BS) can be much higher compared to sub-6GHz wireless systems. Meanwhile, due to the high cost of infrastructure upgrade, it would be difficult for operators to drastically enhance the capacity of backhaul links between mmWave BSs and the core network. As a result, the data rate provided by backhaul may not be sufficient to support all mmWave links, the ba…
Millimeter-wave (mmWave) base station can offer abundant high capacity channel resources toward connected vehicles so that quality-of-service (QoS) of them in terms of downlink throughput can be highly improved. The mmWave base station can operate among existing base stations (e.g., macro-cell base station) on non-overlapped channels among them and the vehicles can make decision what base station to associate, and what channel to utilize on heterogeneous networks. Furthermore, because of the non-omni property of mmWave communication, the vehicles decide how to align the beam direction toward m…
EURECOM（欧洲通信学院）；LTCI, Telecom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院LTCI）
The problem of resource constrained scheduling in a dynamic and heterogeneous wireless setting is considered here. In our setup, the available limited bandwidth resources are allocated in order to serve randomly arriving service demands, which in turn belong to different classes in terms of payload data requirement, delay tolerance, and importance/priority. In addition to heterogeneous traffic, another major challenge stems from random service rates due to time-varying wireless communication channels. Various approaches for scheduling and resource allocation can be used, ranging from simple gr…
Hangzhou Research Center, Huawei Technologies（华为技术有限公司杭州研究中心）；Ottawa Research Center, Huawei Technologies（华为技术有限公司渥太华研究中心）
To improve the system performance towards the Shannon limit, advanced radio resource management mechanisms play a fundamental role. In particular, scheduling should receive much attention, because it allocates radio resources among different users in terms of their channel conditions and QoS requirements. The difficulties of scheduling algorithms are the tradeoffs need to be made among multiple objectives, such as throughput, fairness and packet drop rate. We propose a smart scheduling scheme based on deep reinforcement learning (DRL). We not only verify the performance gain achieved, but also…
University of Electronic Science and Technology of China（电子科技大学）
With the rapid development of deep learning, deep reinforcement learning (DRL) began to appear in the field of resource scheduling in recent years. Based on the previous research on DRL in the literature, we introduce online resource scheduling algorithm DeepRM2 and the offline resource scheduling algorithm DeepRM_Off. Compared with the state-of-the-art DRL algorithm DeepRM and heuristic algorithms, our proposed algorithms have faster convergence speed and better scheduling efficiency with regarding to average slowdown time, job completion time and rewards.
Center for Cyber Cognitive Intelligence (CCI), Fraunhofer IPA（弗劳恩霍夫制造工程与自动化研究所网络认知智能中心）；Institute of Industrial Manufacturing and Management IFF, University of Stuttgart（斯图加特大学工业制造与管理研究所）
Maintenance scheduling is a complex decision-making problem in the production domain, where a number of maintenance tasks and resources has to be assigned and scheduled to production entities in order to prevent unplanned production downtime. Intelligent maintenance strategies are required that are able to adapt to the dynamics and different conditions of production systems. The paper introduces a deep reinforcement learning approach for condition-oriented maintenance scheduling in flow line systems. Different policies are learned, analyzed and evaluated against a benchmark scheduling heuristi…
Energy Efficient Sleep Mode Optimization in 5G mmWave Networks via Multi Agent Deep Reinforcement Learning
Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC（北卡罗来纳州立大学电气与计算机工程系）；Universitat Politècnica de València, Valencia, Spain（瓦伦西亚理工大学）
Singapore University of Technology and Design（新加坡科技设计大学）；Beihang University（北京航空航天大学）
This paper investigates the problem of providing ultra-reliable and energy-efficient virtual reality (VR) experiences for wireless mobile users. To ensure reliable ultra-high-definition (UHD) video frame delivery to mobile users and enhance their immersive visual experiences, a coordinated multipoint (CoMP) transmission technique and millimeter wave (mmWave) communications are exploited. Owing to user movement and time-varying wireless channels, the wireless VR experience enhancement problem is formulated as a sequence-dependent and mixed-integer problem with a goal of maximizing users' feelin…
East China Normal University（华东师范大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）；The Chinese University of Hong Kong-Shenzhen（香港中文大学（深圳））
A novel simulator called VMAgent is introduced to help RL researchers better explore new methods, especially for virtual machine scheduling. VMAgent is inspired by practical virtual machine (VM) scheduling tasks and provides an efficient simulation platform that can reflect the real situations of cloud computing. Three scenarios (fading, recovering, and expansion) are concluded from practical cloud computing and corresponds to many reinforcement learning challenges (high dimensional state and action spaces, high non-stationarity, and life-long demand). VMAgent provides flexible configurations …
CEA-LETI（法国原子能与替代能源委员会电子与信息技术实验室）；Huawei Technologies（华为技术有限公司）；University of Toronto（多伦多大学）
Network densification and millimeter-wave technologies are key enablers to fulfill the capacity and data rate requirements of the fifth generation (5G) of mobile networks. In this context, designing low-complexity policies with local observations, yet able to adapt the user association with respect to the global network state and to the network dynamics is a challenge. In fact, the frameworks proposed in literature require continuous access to global network information and to recompute the association when the radio environment changes. With the complexity associated to such an approach, thes…
We consider the problem of scheduling in constrained queueing networks with a view to minimizing packet delay. Modern communication systems are becoming increasingly complex, and are required to handle multiple types of traffic with widely varying characteristics such as arrival rates and service times. This, coupled with the need for rapid network deployment, render a bottom up approach of first characterizing the traffic and then devising an appropriate scheduling protocol infeasible. In contrast, we formulate a top down approach to scheduling where, given an unknown network and a set of sch…
Research Center of Smart Networks and Systems, Fudan University（复旦大学智慧网络与系统研究中心）；School of Operations Research and Information Engineering, Cornell University（康奈尔大学运筹学与信息工程学院）
Minimizing job scheduling time is a fundamental issue in data center networks that has been extensively studied in recent years. The incoming jobs require different CPU and memory units, and span different number of time slots. The traditional solution is to design efficient heuristic algorithms with performance guarantee under certain assumptions. In this paper, we improve a recently proposed job scheduling algorithm using deep reinforcement learning and extend it to multiple server clusters. Our study reveals that deep reinforcement learning method has the potential to outperform traditional…
In this extended abstract, we propose a new technique for query scheduling with the explicit goal of reducing disk reads and thus implicitly increasing query performance. We introduce SmartQueue, a learned scheduler that leverages overlapping data reads among incoming queries and learns a scheduling strategy that improves cache hits. SmartQueue relies on deep reinforcement learning to produce workload-specific scheduling strategies that focus on long-term performance benefits while being adaptive to previously-unseen data access patterns. We present results from a proof-of-concept prototype, d…
SINTEF Digital（SINTEF数字）；SINTEF Energy Research（SINTEF能源研究）
We explore the use of deep reinforcement learning to provide strategies for long term scheduling of hydropower production. We consider a use-case where the aim is to optimise the yearly revenue given week-by-week inflows to the reservoir and electricity prices. The challenge is to decide between immediate water release at the spot price of electricity and storing the water for later power production at an unknown price, given constraints on the system. We successfully train a soft actor-critic algorithm on a simplified scenario with historical data from the Nordic power market. The presented m…
Illinois Institute of Technology（伊利诺伊理工学院）；Argonne National Laboratory（阿贡国家实验室）；Northern Illinois University（北伊利诺伊大学）
Cluster scheduler is crucial in high-performance computing (HPC). It determines when and which user jobs should be allocated to available system resources. Existing cluster scheduling heuristics are developed by human experts based on their experience with specific HPC systems and workloads. However, the increasing complexity of computing systems and the highly dynamic nature of application workloads have placed tremendous burden on manually designed and tuned scheduling heuristics. More aggressive optimization and automation are needed for cluster scheduling in HPC. In this work, we present a…
Department of Computer Science Texas Tech University（得克萨斯理工大学计算机科学系）
Efficient job scheduling on data centers under heterogeneous complexity is crucial but challenging since it involves the allocation of multi-dimensional resources over time and space. To adapt the complex computing environment in data centers, we proposed an innovative Advantage Actor-Critic (A2C) deep reinforcement learning based approach called A2cScheduler for job scheduling. A2cScheduler consists of two agents, one of which, dubbed the actor, is responsible for learning the scheduling policy automatically and the other one, the critic, reduces the estimation error. Unlike previous policy g…
This paper proposes a safe reinforcement learning algorithm for generation bidding decisions and unit maintenance scheduling in a competitive electricity market environment. In this problem, each unit aims to find a bidding strategy that maximizes its revenue while concurrently retaining its reliability by scheduling preventive maintenance. The maintenance scheduling provides some safety constraints which should be satisfied at all times. Satisfying the critical safety and reliability constraints while the generation units have an incomplete information of each others' bidding strategy is a ch…
Safe Hierarchical Reinforcement Learning for CubeSat Task Scheduling Based on Energy Consumption
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg (UL)（卢森堡大学跨学科安全、可靠性与信任中心（SnT））；Automation and Robotics Research Group (ARG)（自动化与机器人研究组）；Space Systems Research Group (SpaSys)（空间系统研究组）
Millimeter wave (mmWave) bands are considered a powerful key enabler for next generation (5G) mobile networks by providing multi-Gbps data rates. However, their severe pathloss and sensitivity to blockage present challenges in practical implementation. One effective way to mitigate these effects and to increase communication range is beamforming in combination with relaying. In this paper, we focus on two typical mmWave relay networks and for each network, we propose three beam scheduling methods to approach the network information theoretic capacity. The proposed beam scheduling methods inclu…
We are interested in the optimal scheduling of a collection of multi-component application jobs in an edge computing system that consists of geo-distributed edge computing nodes connected through a wide area network. The scheduling and placement of application jobs in an edge system is challenging due to the interdependence of multiple components of each job, and the communication delays between the geographically distributed data sources and edge nodes and their dynamic availability. In this paper we explore the feasibility of applying Deep Reinforcement Learning (DRL) based design to address…
Recent years have seen a rise in interest in terms of using machine learning, particularly reinforcement learning (RL), for production scheduling problems of varying degrees of complexity. The general approach is to break down the scheduling problem into a Markov Decision Process (MDP), whereupon a simulation implementing the MDP is used to train an RL agent. Since existing studies rely on (sometimes) complex simulations for which the code is unavailable, the experiments presented are hard, or, in the case of stochastic environments, impossible to reproduce accurately. Furthermore, there is a …
TCS Research, New Delhi（塔塔咨询服务公司研究部（TCS Research），新德里）
In this paper we present Meeting Bot, a reinforcement learning based conversational system that interacts with multiple users to schedule meetings. The system is able to interpret user utterences and map them to preferred time slots, which are then fed to a reinforcement learning (RL) system with the goal of converging on an agreeable time slot. The RL system is able to adapt to user preferences and environmental changes in meeting arrival rate while still scheduling effectively. Learning is performed via policy gradient with exploration, by utilizing an MLP as an approximator of the policy fu…
Energy consumption is one of the most critical concerns in designing computing devices, ranging from portable embedded systems to computer cluster systems. Furthermore, in the past decade, cluster systems have increasingly risen as popular platforms to run computing-intensive real-time applications in which the performance is of great importance. However, due to different characteristics of real-time workloads, developing general job scheduling solutions that efficiently address both energy consumption and performance in real-time cluster systems is a challenging problem. In this paper, inspir…
Reinforcement Learning for Adaptive Resource Scheduling in Complex System Environments
Northeastern University（东北大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Utah（犹他大学）；Columbia University（哥伦比亚大学）；Western University（韦仕敦大学）
RF Network parametric optimization requires a wealth of experience and knowledge to achieve the optimal balance between coverage, capacity, system efficiency and customer experience from the telecom sites serving the users. With 5G, the complications of Air interface scheduling have increased due to the usage of massive MIMO, beamforming and introduction of higher modulation schemes with varying numerologies. In this work, we tune a machine learning model to "learn" the best combination of parameters for a given traffic profile using Cross Entropy Method Reinforcement Learning and compare thes…
Scheduling plays a pivotal role in multi-user wireless communications, since the quality of service of various users largely depends upon the allocated radio resources. In this paper, we propose a novel scheduling algorithm with contiguous frequency-domain resource allocation (FDRA) based on deep reinforcement learning (DRL) that jointly selects users and allocates resource blocks (RBs). The scheduling problem is modeled as a Markov decision process, and a DRL agent determines which user and how many consecutive RBs for that user should be scheduled at each RB allocation step. The state space,…
We propose a novel user equipment (UE) scheduling framework for millimeter-wave (mmWave) massive multiuser (MU) multiple-input multiple-output (MIMO) wireless systems. Our framework determines (sub)sets of UEs that should transmit simultaneously in a given time slot by approximately solving a nonconvex optimization problem using forward-backward splitting. Our UE scheduling framework is flexible in the sense that it (i) supports a variety of cost functions, including post-equalization mean square error and sum rate, and (ii) enables precise control over the minimum and maximum number of resour…
Northeast Electric Power University（东北电力大学）；Huazhong University of Science and Technology（华中科技大学）；Cranfield University（克兰菲尔德大学）
Multi-uncertainties from power sources and loads have brought significant challenges to the stable demand supply of various resources at islands. To address these challenges, a comprehensive scheduling framework is proposed by introducing a model-free deep reinforcement learning (DRL) approach based on modeling an island integrated energy system (IES). In response to the shortage of freshwater on islands, in addition to the introduction of seawater desalination systems, a transmission structure of "hydrothermal simultaneous transmission" (HST) is proposed. The essence of the IES scheduling pro…
University of Klagenfurt（克拉根福大学）；Graz University of Technology（格拉茨技术大学）
Scheduling is a fundamental task occurring in various automated systems applications, e.g., optimal schedules for machines on a job shop allow for a reduction of production costs and waste. Nevertheless, finding such schedules is often intractable and cannot be achieved by Combinatorial Optimization Problem (COP) methods within a given time limit. Recent advances of Deep Reinforcement Learning (DRL) in learning complex behavior enable new COP application possibilities. This paper presents an efficient DRL environment for Job-Shop Scheduling -- an important problem in the field. Furthermore, we…
US Air Force-MIT AI Accelerator（美国空军-麻省理工学院人工智能加速器）；MIT（麻省理工学院）
Integer programs provide a powerful abstraction for representing a wide range of real-world scheduling problems. Despite their ability to model general scheduling problems, solving large-scale integer programs (IP) remains a computational challenge in practice. The incorporation of more complex objectives such as robustness to disruptions further exacerbates the computational challenge. We present NICE (Neural network IP Coefficient Extraction), a novel technique that combines reinforcement learning and integer programming to tackle the problem of robust scheduling. More specifically, NICE use…
We propose ScheduleNet, a RL-based real-time scheduler, that can solve various types of multi-agent scheduling problems. We formulate these problems as a semi-MDP with episodic reward (makespan) and learn ScheduleNet, a decentralized decision-making policy that can effectively coordinate multiple agents to complete tasks. The decision making procedure of ScheduleNet includes: (1) representing the state of a scheduling problem with the agent-task graph, (2) extracting node embeddings for agent and tasks nodes, the important relational information among agents and tasks, by employing the type-aw…
