DL-DRL: A double-level deep reinforcement learning approach for large-scale task scheduling of multi-UAV
Central South University（中南大学）；Singapore Management University（新加坡管理大学）；University of Alberta（阿尔伯塔大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；Polish Academy of Sciences（波兰科学院）
Exploiting unmanned aerial vehicles (UAVs) to execute tasks is gaining growing popularity recently. To solve the underlying task scheduling problem, the deep reinforcement learning (DRL) based methods demonstrate notable advantage over the conventional heuristics as they rely less on hand-engineered rules. However, their decision space will become prohibitively huge as the problem scales up, thus deteriorating the computation efficiency. To alleviate this issue, we propose a double-level deep reinforcement learning (DL-DRL) approach based on a divide and conquer framework (DCF), where we decompose the task scheduling of multi-UAV into task allocation and route planning. Particularly, we design an encoder-decoder structured policy network in our upper-level DRL model to allocate the tasks to different UAVs, and we exploit another attention based policy network in our lower-level DRL model to construct the route for each UAV, with the objective to maximize the number of executed tasks given the maximum flight distance of the UAV. To effectively train the two models, we design an interactive training strategy (ITS), which includes pre-training, intensive training and alternate training. Experimental results show that our DL-DRL performs favorably against the learning-based and conventional baselines including the OR-Tools, in terms of solution quality and computation efficiency. We also verify the generalization performance of our approach by applying it to larger sizes of up to 1000 tasks. Moreover, we also show via an ablation study that our ITS can help achieve a balance between the performance and training efficiency.
Miami University（迈阿密大学）；Illinois Institute of Technology（伊利诺伊理工学院）
Energy-aware control for multiple unmanned aerial vehicles (UAVs) is one of the major research interests in UAV based networking. Yet few existing works have focused on how the network should react around the timing when the UAV lineup is changed. In this work, we study proactive self-remedy of energy-constrained UAV networks when one or more UAVs are short of energy and about to quit for charging. We target at an energy-aware optimal UAV control policy which proactively relocates the UAVs when any UAV is about to quit the network, rather than passively dispatches the remaining UAVs after the …
Research Institute of Electrical and Computer Engineering, Hanyang University（汉阳大学电气与计算机工程研究所）；Hanyang University（汉阳大学）
In this paper, we propose SACHER (soft actor-critic (SAC) with hindsight experience replay (HER)), which constitutes a class of deep reinforcement learning (DRL) algorithms. SAC is known as an off-policy model-free DRL algorithm based on the maximum entropy framework, which outperforms earlier DRL algorithms in terms of exploration, robustness and learning performance. However, in SAC, maximizing the entropy-augmented objective may degrade the optimality of learning outcomes. HER is known as a sample-efficient replay method that enhances the performance of off-policy DRL algorithms by allowing…
Scheduling plays a pivotal role in multi-user wireless communications, since the quality of service of various users largely depends upon the allocated radio resources. In this paper, we propose a novel scheduling algorithm with contiguous frequency-domain resource allocation (FDRA) based on deep reinforcement learning (DRL) that jointly selects users and allocates resource blocks (RBs). The scheduling problem is modeled as a Markov decision process, and a DRL agent determines which user and how many consecutive RBs for that user should be scheduled at each RB allocation step. The state space,…
DRL-Enabled Trajectory Planing for UAV-Assisted VLC: Optimal Altitude and Reward Design
Department of Information and Communication Engineering, Tongji University（信息与通信工程系，同济大学）；Department of Electronic Engineering, Shanghai Jiao Tong University（电子工程系，上海交通大学）；Shenzhen International Graduate School, Tsinghua University（深圳国际研究生院，清华大学）
University of Electronic Science and Technology of China（电子科技大学）
With the rapid development of deep learning, deep reinforcement learning (DRL) began to appear in the field of resource scheduling in recent years. Based on the previous research on DRL in the literature, we introduce online resource scheduling algorithm DeepRM2 and the offline resource scheduling algorithm DeepRM_Off. Compared with the state-of-the-art DRL algorithm DeepRM and heuristic algorithms, our proposed algorithms have faster convergence speed and better scheduling efficiency with regarding to average slowdown time, job completion time and rewards.
Deep Reinforcement Learning (DRL) is gaining attention as a potential approach to design trajectories for autonomous unmanned aerial vehicles (UAV) used as flying access points in the context of cellular or Internet of Things (IoT) connectivity. DRL solutions offer the advantage of on-the-go learning hence relying on very little prior contextual information. A corresponding drawback however lies in the need for many learning episodes which severely restricts the applicability of such approach in real-world time- and energy-constrained missions. Here, we propose a model-aided deep Q-learning ap…
Structure-Enhanced DRL for Optimal Transmission Scheduling
School of Electrical and Information Engineering, The University of Sydney（悉尼大学电气与信息工程学院）；School of Electrical Engineering and Robotics, Queensland University of Technology (QUT)（昆士兰科技大学电气与机器人工程学院）；Nokia Bell Laboratories（诺基亚贝尔实验室）
Delft University of Technology（代尔夫特理工大学）；Eindhoven University of Technology（埃因霍温理工大学）
Taking advantage of their data-driven and model-free features, Deep Reinforcement Learning (DRL) algorithms have the potential to deal with the increasing level of uncertainty due to the introduction of renewable-based generation. To deal simultaneously with the energy systems' operational cost and technical constraints (e.g, generation-demand power balance) DRL algorithms must consider a trade-off when designing the reward function. This trade-off introduces extra hyperparameters that impact the DRL algorithms' performance and capability of providing feasible solutions. In this paper, a perfo…
Northwestern Polytechnical University（西北工业大学）；Cranfield University（克兰菲尔德大学）；City, University of London（伦敦大学城市学院）
In this paper, a deep reinforcement learning (DRL) method is proposed to address the problem of UAV navigation in an unknown environment. However, DRL algorithms are limited by the data efficiency problem as they typically require a huge amount of data before they reach a reasonable performance. To speed up the DRL training process, we developed a novel learning framework which combines imitation learning and reinforcement learning and building upon Twin Delayed DDPG (TD3) algorithm. We newly introduced both policy and Q-value network are learned using the expert demonstration during the imita…
University of Saskatchewan（萨斯喀彻温大学）；Cisco Systems Inc.（思科系统公司）
Employing unmanned aerial vehicles (UAVs) has attracted growing interests and emerged as the state-of-the-art technology for data collection in Internet-of-Things (IoT) networks. In this paper, with the objective of minimizing the total energy consumption of the UAV-IoT system, we formulate the problem of jointly designing the UAV's trajectory and selecting cluster heads in the IoT network as a constrained combinatorial optimization problem which is classified as NP-hard and challenging to solve. We propose a novel deep reinforcement learning (DRL) with a sequential model strategy that can eff…
UAV-Assisted Resilience in 6G and Beyond Network Energy Saving: A Multi-Agent DRL Approach
National Economics University（国立经济大学）；Norwegian University of Science and Technology（挪威科技大学）；Waseda University（早稻田大学）；Duy Tan University（杜维谭大学）；University of Luxembourg（卢森堡大学）；National Academy of Education Management（国家教育管理学院）
Personalized Federated Deep Reinforcement Learning-based Trajectory Optimization for Multi-UAV Assisted Edge Computing
Nanjing University of Science and Technology（南京理工大学）；Zhejiang Lab（之江实验室）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）；ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC联合学院）；Louisiana State University（路易斯安那州立大学）
Hangzhou Research Center, Huawei Technologies（华为技术有限公司杭州研究中心）；Ottawa Research Center, Huawei Technologies（华为技术有限公司渥太华研究中心）
To improve the system performance towards the Shannon limit, advanced radio resource management mechanisms play a fundamental role. In particular, scheduling should receive much attention, because it allocates radio resources among different users in terms of their channel conditions and QoS requirements. The difficulties of scheduling algorithms are the tradeoffs need to be made among multiple objectives, such as throughput, fairness and packet drop rate. We propose a smart scheduling scheme based on deep reinforcement learning (DRL). We not only verify the performance gain achieved, but also…
Virginia Tech（弗吉尼亚理工大学）；University of Colorado Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）；Ericsson Research（爱立信研究院）；Princeton University（普林斯顿大学）；University of Oulu（奥卢大学）
Unmanned aerial vehicle (UAV) swarms must exploit machine learning (ML) in order to execute various tasks ranging from coordinated trajectory planning to cooperative target recognition. However, due to the lack of continuous connections between the UAV swarm and ground base stations (BSs), using centralized ML will be challenging, particularly when dealing with a large volume of data. In this paper, a novel framework is proposed to implement distributed federated learning (FL) algorithms within a UAV swarm that consists of a leading UAV and several following UAVs. Each following UAV trains a l…
In unmanned aerial vehicle (UAV) applications, the UAV's limited energy supply and storage have triggered the development of intelligent energy-conserving scheduling solutions. In this paper, we investigate energy minimization for UAV-aided communication networks by jointly optimizing data-transmission scheduling and UAV hovering time. The formulated problem is combinatorial and non-convex with bilinear constraints. To tackle the problem, firstly, we provide an optimal relax-and-approximate solution and develop a near-optimal algorithm. Both the proposed solutions are served as offline perform…
Structure-Enhanced Deep Reinforcement Learning for Optimal Transmission Scheduling
The University of Sydney（悉尼大学）；School of Electrical and Information Engineering, The University of Sydney（悉尼大学电气与信息工程学院）；Queensland University of Technology（昆士兰科技大学）；School of Electrical Engineering and Robotics, Queensland University of Technology（昆士兰科技大学电气工程与机器人学院）
University of Waterloo（滑铁卢大学）；Zhejiang University（浙江大学）；Huazhong University of Science and Technology（华中科技大学）；Central South University（中南大学）；Xidian University（西安电子科技大学）
In this paper, we investigate a computing task scheduling problem in space-air-ground integrated network (SAGIN) for delay-oriented Internet of Things (IoT) services. In the considered scenario, an unmanned aerial vehicle (UAV) collects computing tasks from IoT devices and then makes online offloading decisions, in which the tasks can be processed at the UAV or offloaded to the nearby base station or the remote satellite. Our objective is to design a task scheduling policy that minimizes offloading and computing delay of all tasks given the UAV energy capacity constraint. To this end, we first…
University of Klagenfurt（克拉根福大学）；Graz University of Technology（格拉茨技术大学）
Scheduling is a fundamental task occurring in various automated systems applications, e.g., optimal schedules for machines on a job shop allow for a reduction of production costs and waste. Nevertheless, finding such schedules is often intractable and cannot be achieved by Combinatorial Optimization Problem (COP) methods within a given time limit. Recent advances of Deep Reinforcement Learning (DRL) in learning complex behavior enable new COP application possibilities. This paper presents an efficient DRL environment for Job-Shop Scheduling -- an important problem in the field. Furthermore, we…
LLM-Enabled In-Context Learning for Data Collection Scheduling in UAV-assisted Sensor Networks
Real-Time and Embedded Computing Systems Research Centre (CISTER)（实时与嵌入式计算系统研究中心（CISTER））；Faculdade de Engenharia da Universidade do Porto（葡萄牙波尔图大学工程学院）；School of Computer Science, McGill University（麦吉尔大学计算机科学学院）；Knight Foundation School of Computing and Information Sciences, Florida International University（国际大学计算与信息科学学院（Knight基金会））
College of Intelligence Science and Technology, National University of Defense Technology（国防科技大学智能科学与技术学院）
Developing the flocking behavior for a dynamic squad of fixed-wing UAVs is still a challenge due to kinematic complexity and environmental uncertainty. In this paper, we deal with the decentralized flocking and collision avoidance problem through deep reinforcement learning (DRL). Specifically, we formulate a decentralized DRL-based decision making framework from the perspective of every follower, where a collision avoidance mechanism is integrated into the flocking controller. Then, we propose a novel reinforcement learning algorithm PS-CACER for training a shared control policy for all the f…
Hunan Normal University（湖南师范大学）；Northumbria University（诺森比亚大学）；Hunan University of Technology and Business（湖南工商大学）；Queen Mary University of London（伦敦玛丽女王大学）；Changchun Institute of Technology（长春工程学院）；University of Essex（埃塞克斯大学）
An online resource scheduling framework is proposed for minimizing the sum of weighted task latency for all the Internet of things (IoT) users, by optimizing offloading decision, transmission power and resource allocation in the large-scale mobile edge computing (MEC) system. Towards this end, a deep reinforcement learning (DRL) based solution is proposed, which includes the following components. Firstly, a related and regularized stacked auto encoder (2r-SAE) with unsupervised learning is applied to perform data compression and representation for high dimensional channel quality information (…
Multi-objective task scheduling (MOTS) is the task scheduling while optimizing multiple and possibly contradicting constraints. A challenging extension of this problem occurs when every individual task is a multi-objective optimization problem by itself. While deep reinforcement learning (DRL) has been successfully applied to complex sequential problems, its application to the MOTS domain has been stymied by two challenges. The first challenge is the inability of the DRL algorithm to ensure that every item is processed identically regardless of its position in the queue. The second challenge i…
In this paper, the downlink packet scheduling problem for cellular networks is modeled, which jointly optimizes throughput, fairness and packet drop rate. Two genie-aided heuristic search methods are employed to explore the solution space. A deep reinforcement learning (DRL) framework with A2C algorithm is proposed for the optimization problem. Several methods have been utilized in the framework to improve the sampling and training efficiency and to adapt the algorithm to a specific scheduling problem. Numerical results show that DRL outperforms the baseline algorithm and achieves similar perf…
An Agentic AI Framework with Large Language Models and Chain-of-Thought for UAV-Assisted Logistics Scheduling with Mobile Edge Computing
Nanyang Technological University（南洋理工大学）；Singapore Institute of Technology（新加坡理工学院）；Seatrium New Energy Laboratory（Seatrium 新能源实验室）；Ministry of Education (MOE) Tier 1（教育部 Tier 1）；Research Innovation and Enterprise (RIE) 2025 Industry Alignment Fund-Industry Collaboration Projects (IAF-ICP)（研究创新与企业 (RIE) 2025 行业对齐基金-行业合作项目 (IAF-ICP)）
Integrating artificial intelligence (AI) into wireless networks has drawn significant interest in both industry and academia. A common solution is to replace partial or even all modules in the conventional systems, which is often lack of efficiency and robustness due to their ignoring of expert knowledge. In this paper, we take deep reinforcement learning (DRL) based scheduling as an example to investigate how expert knowledge can help with AI module in cellular networks. A simulation platform, which has considered link adaption, feedback and other practical mechanisms, is developed to facilit…
Miami University（迈阿密大学）；Illinois Institute of Technology（伊利诺伊理工学院）；University of Waterloo（滑铁卢大学）
In this chapter, the regulation of Unmanned Aerial Vehicle (UAV) communication network is investigated in the presence of dynamic changes in the UAV lineup and user distribution. We target an optimal UAV control policy which is capable of identifying the upcoming change in the UAV lineup (quit or join-in) or user distribution, and proactively relocating the UAVs ahead of the change rather than passively dispatching the UAVs after the change. Specifically, a deep reinforcement learning (DRL)-based UAV control framework is developed to maximize the accumulated user satisfaction (US) score for a …
Indian Institute of Science（印度科学学院）；Missouri Univ. of Science & Tech.（密苏里科技大学）
Unmanned Aerial Vehicles (UAVs) or drones are increasingly used for urban applications like traffic monitoring and construction surveys. Autonomous navigation allows drones to visit waypoints and accomplish activities as part of their mission. A common activity is to hover and observe a location using on-board cameras. Advances in Deep Neural Networks (DNNs) allow such videos to be analyzed for automated decision making. UAVs also host edge computing capability for on-board inferencing by such DNNs. To this end, for a fleet of drones, we propose a novel Mission Scheduling Problem (MSP) that co…
University of Electronic Science and Technology of China（电子科技大学）；Institute for Infocomm Research, Agency for Science, Technology and Research（新加坡科技研究局信息通信研究院）
This paper investigates a master unmanned aerial vehicle (MUAV)-powered Internet of Things (IoT) network, in which we propose using a rechargeable auxiliary UAV (AUAV) equipped with an intelligent reflecting surface (IRS) to enhance the communication signals from the MUAV and also leverage the MUAV as a recharging power source. Under the proposed model, we investigate the optimal collaboration strategy of these energy-limited UAVs to maximize the accumulated throughput of the IoT network. Depending on whether there is charging between the two UAVs, two optimization problems are formulated. To …
RESCHED: Rethinking Flexible Job Shop Scheduling from a Transformer-based Architecture with Simplified States
School of Computing and Information Systems, Singapore Management University, Singapore（新加坡管理大学计算机与信息系）；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院）；Institute of Marine Science and Technology, Shandong University, China（山东大学海洋科学与技术研究院）
We are interested in the optimal scheduling of a collection of multi-component application jobs in an edge computing system that consists of geo-distributed edge computing nodes connected through a wide area network. The scheduling and placement of application jobs in an edge system is challenging due to the interdependence of multiple components of each job, and the communication delays between the geographically distributed data sources and edge nodes and their dynamic availability. In this paper we explore the feasibility of applying Deep Reinforcement Learning (DRL) based design to address…
Deep progressive reinforcement learning-based flexible resource scheduling framework for IRS and UAV-assisted MEC system
School of Computer Science, Hunan University of Technology and Business（湖南工商大学计算机学院）；Xiangjiang Laboratory（湘江实验室）；Hunan Normal University（湖南师范大学）；College of Information Science and Engineering, Hunan Normal University（湖南师范大学信息科学与工程学院）
