Placement Optimization is an important problem in systems and chip design, which consists of mapping the nodes of a graph onto a limited set of resources to optimize for an objective, subject to constraints. In this paper, we start by motivating reinforcement learning as a solution to the placement problem. We then give an overview of what deep reinforcement learning is. We next formulate the placement problem as a reinforcement learning problem and show how this problem can be solved with policy gradient optimization. Finally, we describe lessons we have learned from training deep reinforcement learning policies across a variety of placement optimization problems.
Recently, successful applications of reinforcement learning to chip placement have emerged. Pretrained models are necessary to improve efficiency and effectiveness. Currently, the weights of objective metrics (e.g., wirelength, congestion, and timing) are fixed during pretraining. However, fixed-weighed models cannot generate the diversity of placements required for engineers to accommodate changing requirements as they arise. This paper proposes flexible multiple-objective reinforcement learning (MORL) to support objective functions with inference-time variable weights using just a single pre…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google（谷歌）
In physical design, human designers typically place macros via trial and error, which is a Markov decision process. Reinforcement learning (RL) methods have demonstrated superhuman performance on the macro placement. In this paper, we propose an extension to this prior work (Mirhoseini et al., 2020). We first describe the details of the policy and value network architecture. We replace the force-directed method with DREAMPlace for placing standard cells in the RL environment. We also compare our improved method with other academic placers on public benchmarks.
Physical design and production of Integrated Circuits (IC) is becoming increasingly more challenging as the sophistication in IC technology is steadily increasing. Placement has been one of the most critical steps in IC physical design. Through decades of research, partition-based, analytical-based and annealing-based placers have been enriching the placement solution toolbox. However, open challenges including long run time and lack of ability to generalize continue to restrict wider applications of existing placement tools. We devise a learning-based placement tool based on cyclic applicatio…
In this work, we present a learning-based approach to chip placement, one of the most complex and time-consuming stages of the chip design process. Unlike prior methods, our approach has the ability to learn from past experience and improve over time. In particular, as we train over a greater number of chip blocks, our method becomes better at rapidly generating optimized placements for previously unseen chip blocks. To achieve these results, we pose placement as a Reinforcement Learning (RL) problem and train an agent to place the nodes of a chip netlist onto a chip canvas. To enable our RL p…
For its advantage in GPU acceleration and less dependency on human experts, machine learning has been an emerging tool for solving the placement and routing problems, as two critical steps in modern chip design flow. Being still in its early stage, there are fundamental issues: scalability, reward design, and end-to-end learning paradigm etc. To achieve end-to-end placement learning, we first propose a joint learning method termed by DeepPlace for the placement of macros and standard cells, by the integration of reinforcement learning with a gradient based optimization scheme. To further bridg…
Google Brain（谷歌大脑）；Google Brain Residency Program（谷歌大脑驻留计划）；Google（谷歌）
The past few years have witnessed a growth in size and computational requirements for training and inference with neural networks. Currently, a common approach to address these requirements is to use a heterogeneous distributed environment with a mixture of hardware devices such as CPUs and GPUs. Importantly, the decision of placing parts of the neural models on devices is often made by human experts based on simple heuristics and intuitions. In this paper, we propose a method which learns to optimize device placement for TensorFlow computational graphs. Key to our method is the use of a seque…
For deep neural network accelerators, memory movement is both energetically expensive and can bound computation. Therefore, optimal mapping of tensors to memory hierarchies is critical to performance. The growing complexity of neural networks calls for automated memory mapping instead of manual heuristic approaches; yet the search space of neural network computational graphs have previously been prohibitively large. We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces, that combines graph neural networks, reinforcement learning, and evolution…
Concordia Institute for Information Systems Engineering, Concordia University（康考迪亚大学康考迪亚信息系统工程研究所）；Lebanese American University（黎巴嫩美国大学）
Cloud computing is a reliable solution to provide distributed computation power. However, real-time response is still challenging regarding the enormous amount of data generated by the IoT devices in 5G and 6G networks. Thus, multi-access edge computing (MEC), which consists of distributing the edge servers in the proximity of end-users to have low latency besides the higher processing power, is increasingly becoming a vital factor for the success of modern applications. This paper addresses the problem of minimizing both, the network delay, which is the main objective of MEC, and the number o…
Beijing University of Posts and Telecommunications（北京邮电大学）；SUTD（新加坡科技设计大学）；Commonwealth Scientific and Industrial Research Organisation (CSIRO)（联邦科学与工业研究组织（CSIRO））；NTU（南洋理工大学）
This paper proposes an effective and novel multiagent deep reinforcement learning (MADRL)-based method for solving the joint virtual network function (VNF) placement and routing (P&R), where multiple service requests with differentiated demands are delivered at the same time. The differentiated demands of the service requests are reflected by their delay- and cost-sensitive factors. We first construct a VNF P&R problem to jointly minimize a weighted sum of service delay and resource consumption cost, which is NP-complete. Then, the joint VNF P&R problem is decoupled into two iterative subtasks…
Deep Reinforcement Learning for Urban Air Quality Management: Multi-Objective Optimization of Pollution Mitigation Booth Placement in Metropolitan Environments
School of Computer Science（计算机科学学院）；Engineering, Vellore Institute of Technology（工程学院，维洛雷理工学院）；Chennai Campus（钦奈校区）；Chennai, Tamil Nadu 600127, India（钦奈，泰米尔纳德邦600127，印度）
Advancing Forest Fire Prevention: Deep Reinforcement Learning for Effective Firebreak Placement
University of Chile（智利大学）；Universidad Tecnológica Metropolitana（大都会技术大学）；Complex Engineering System Institute - ISCI（复杂工程系统研究所）；Rey Juan Carlos University（胡安卡洛斯国王大学）；Centre de Ciència i Tecnologia Forestal de Catalunya（加泰罗尼亚森林科学与技术中心）
We propose to demonstrate a network slice placement optimization solution based on Deep Reinforcement Learning (DRL), referred to as Heuristically-controlled DRL, which uses a heuristic to control the DRL algorithm convergence. The solution is adapted to realistic networks with large scale and under non-stationary traffic conditions (namely, the network load). We demonstrate the applicability of the proposed solution and its higher and stable performance over a non-controlled DRL-based solution. Demonstration scenarios include full online learning with multiple volatile network slice placement…
Unmanned aerial vehicles (UAVs) can be utilized as aerial base stations (ABSs) to assist terrestrial infrastructure for keeping wireless connectivity in various emergency scenarios. To maximize the coverage rate of N ground users (GUs) by jointly placing multiple ABSs with limited coverage range is known to be a NP-hard problem with exponential complexity in N. The problem is further complicated when the coverage range becomes irregular due to site-specific blockage (e.g., buildings) on the air-ground channel in the 3-dimensional (3D) space. To tackle this challenging problem, this paper appli…
NTT Communication Science Laboratories（NTT通信科学实验室）；Institute of Mathematics for Industry, Kyushu University（九州大学数学工学研究所）；Center for Advanced Intelligence Project, RIKEN（理化学研究所高级智能项目中心）
We propose a data-driven method for controlling the frequency and convergence rate of black-box nonlinear dynamical systems based on the Koopman operator theory. With the proposed method, a policy network is trained such that the eigenvalues of a Koopman operator of controlled dynamics are close to the target eigenvalues. The policy network consists of a neural network to find a Koopman invariant subspace, and a pole placement module to adjust the eigenvalues of the Koopman operator. Since the policy network is differentiable, we can train it in an end-to-end fashion using reinforcement learni…
Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）
Machine learning (ML) has made incredible impacts and transformations in a wide range of vehicular applications. As the use of ML in Internet of Vehicles (IoV) continues to advance, adversarial threats and their impact have become an important subject of research worth exploring. In this paper, we focus on Sybil-based adversarial threats against a deep reinforcement learning (DRL)-assisted IoV framework and more specifically, DRL-based dynamic service placement in IoV. We carry out an experimental study with real vehicle trajectories to analyze the impact on service delay and resource congesti…
TDPNavigator-Placer: Thermal- and Wirelength-Aware Chiplet Placement in 2.5D Systems Through Multi-Agent Reinforcement Learning
Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR）信息通信研究院（I2R））；Institute of Microelectronics (IME), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR）微电子研究院（IME））
Comparison of metaheuristics for the firebreak placement problem: a simulation-based optimization approach
University of Chile（智利大学）；Complex Engineering System Institute - ISCI（复杂工程系统研究所（ISCI））；Universidad Tecnológica Metropolitana（都会理工大学）；Universidad de Santiago de Chile（智利圣地亚哥大学）
INSPIRE-GNN: Intelligent Sensor Placement to Improve Sparse Bicycling Network Prediction via Reinforcement Learning Boosted Graph Neural Networks
School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia（公共卫生与预防医学学院，墨尔本大学）；Department of Electrical and Computer System Engineering, Monash University, Melbourne, Australia（电气与计算机系统工程系，墨尔本大学）；School of Civil and Environmental Engineering, UNSW, Sydney, Australia（土木与环境工程学院，新南威尔士大学）；School of Information and Communication Technology, Griffith University, Brisbane, Australia（信息与通信技术学院，格里菲斯大学）
China University of Petroleum (East China)（中国石油大学（华东））；Memorial University of Newfoundland（纽芬兰纪念大学）
Determining optimal well placements and controls are two important tasks in oil field development. These problems are computationally expensive, nonconvex, and contain multiple optima. The practical solution of these problems require efficient and robust algorithms. In this paper, the multilevel coordinate search (MCS) algorithm is applied for well placement and control optimization problems. MCS is a derivative-free algorithm that combines global and local search. Both synthetic and real oil fields are considered. The performance of MCS is compared to generalized pattern search (GPS), particl…
Effective placement of charging stations plays a key role in Electric Vehicle (EV) adoption. In the placement problem, given a set of candidate sites, an optimal subset needs to be selected with respect to the concerns of both (a) the charging station service provider, such as the demand at the candidate sites and the budget for deployment, and (b) the EV user, such as charging station reachability and short waiting times at the station. This work addresses these concerns, making the following three novel contributions: (i) a supervised multi-view learning framework using Canonical Correlation…
BBOPlace-Bench: Benchmarking Black-Box Optimization for Chip Placement
National Key Laboratory for Novel Software Technology（国家新型软件技术重点实验室）；School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）
Free-Placement Optimization of Ground Station Locations for Low-Earth Orbit Satellites
Department of Aeronautics and Astronautics, Stanford, CA（航空航天系，斯坦福大学，加州斯坦福）；Department of Computer Science, Stanford University, Stanford, CA（计算机科学系，斯坦福大学，加州斯坦福）
