Graph neural architecture search has received a lot of attention as Graph Neural Networks (GNNs) has been successfully applied on the non-Euclidean data recently. However, exploring all possible GNNs architectures in the huge search space is too time-consuming or impossible for big graph data. In this paper, we propose a parallel graph architecture search (GraphPAS) framework for graph neural networks. In GraphPAS, we explore the search space in parallel by designing a sharing-based evolution learning, which can improve the search efficiency without losing the accuracy. Additionally, architecture information entropy is adopted dynamically for mutation selection probability, which can reduce space exploration. The experimental result shows that GraphPAS outperforms state-of-art models with efficiency and accuracy simultaneously.
Competence Center for High Performance Computing Fraunhofer ITWM Kaiserslautern（弗劳恩霍夫研究所ITWM高性能计算能力中心（凯泽斯劳滕））；Institute for Machine Learning and Analytics (IMLA) Offenburg University（奥芬堡大学机器学习与分析研究所（IMLA））
We introduce an open source python framework named PHS - Parallel Hyperparameter Search to enable hyperparameter optimization on numerous compute instances of any arbitrary python function. This is achieved with minimal modifications inside the target function. Possible applications appear in expensive to evaluate numerical computations which strongly depend on hyperparameters such as machine learning. Bayesian optimization is chosen as a sample efficient method to propose the next query set of parameters.
Monte Carlo Tree Search (MCTS) has proven to be capable of solving challenging tasks in domains such as Go, chess and Atari. Previous research has developed parallel versions of MCTS, exploiting today's multiprocessing architectures. These studies focused on versions of MCTS for the discrete case. Our work builds upon existing parallelization strategies and extends them to continuous domains. In particular, leaf parallelization and root parallelization are studied and two final selection strategies that are required to handle continuous states in root parallelization are proposed. The evaluati…
Nagoya Institute of Technology（名古屋工业大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Osaka University（大阪大学）；Japan Fine Ceramics Center（日本精细陶瓷中心）
Max-value entropy search (MES) is one of the state-of-the-art approaches in Bayesian optimization (BO). In this paper, we propose a novel variant of MES for constrained problems, called Constrained MES via Information lower BOund (CMES-IBO), that is based on a Monte Carlo (MC) estimator of a lower bound of a mutual information (MI). Unlike existing studies, our MI is defined so that uncertainty with respect to feasibility can be incorporated. We derive a lower bound of the MI that guarantees non-negativity, while a constrained counterpart of conventional MES can be negative. We further provide…
Department of Intelligent Systems, TU Delft（代尔夫特理工大学智能系统系）；Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）；Trent AI Limited（Trent AI有限公司）；Information Systems, TU Eindhoven（埃因霍温理工大学信息系统系）；Centrum Wiskunde & Informatica, Amsterdam（阿姆斯特丹数学与信息学研究中心）
McGill University（麦吉尔大学）；Hydro-Québec’s Research Institute（魁北克水电公司研究院）
We consider computationally expensive blackbox optimization problems and present a method that employs surrogate models and concurrent computing at the search step of the mesh adaptive direct search (MADS) algorithm. Specifically, we solve a surrogate optimization problem using locally weighted scatterplot smoothing (LOWESS) models to find promising candidate points to be evaluated by the blackboxes. We consider several methods for selecting promising points from a large number of points. We conduct numerical experiments to assess the performance of the modified MADS algorithm with respect to …
We develop parallel predictive entropy search (PPES), a novel algorithm for Bayesian optimization of expensive black-box objective functions. At each iteration, PPES aims to select a batch of points which will maximize the information gain about the global maximizer of the objective. Well known strategies exist for suggesting a single evaluation point based on previous observations, while far fewer are known for selecting batches of points to evaluate in parallel. The few batch selection schemes that have been studied all resort to greedy methods to compute an optimal batch. To the best of our…
We present Parallel Feasible Pareto Frontier Entropy Search ($\{\text{PF}\}^2$ES) -- a novel information-theoretic acquisition function for multi-objective Bayesian optimization supporting unknown constraints and batch query. Due to the complexity of characterizing the mutual information between candidate evaluations and (feasible) Pareto frontiers, existing approaches must either employ crude approximations that significantly hamper their performance or rely on expensive inference schemes that substantially increase the optimization's computational overhead. By instead using a variational low…
ParBalans: Parallel Multi-Armed Bandits-based Adaptive Large Neighborhood Search
Department of Mechanical & Industrial Engineering, Northeastern University（东北大学机械与工业工程系）；AI Center of Excellence, Fidelity Investments（fidelity investments 精英研究中心）；Department of Computer Science, University of Southern California（南加州大学计算机科学系）；Department of Computer Science, Brown University（布朗大学计算机科学系）
We present a simple and powerful algorithm for parallel black box optimization called Successive Halving and Classification (SHAC). The algorithm operates in $K$ stages of parallel function evaluations and trains a cascade of binary classifiers to iteratively cull the undesirable regions of the search space. SHAC is easy to implement, requires no tuning of its own configuration parameters, is invariant to the scale of the objective function and can be built using any choice of binary classifier. We adopt tree-based classifiers within SHAC and achieve competitive performance against several str…
University of Luxembourg（卢森堡大学）；Technische Universität Darmstadt（达姆施塔特工业大学）
In this paper, we propose a convergent parallel best-response algorithm with the exact line search for the nondifferentiable nonconvex sparsity-regularized rank minimization problem. On the one hand, it exhibits a faster convergence than subgradient algorithms and block coordinate descent algorithms. On the other hand, its convergence to a stationary point is guaranteed, while ADMM algorithms only converge for convex problems. Furthermore, the exact line search procedure in the proposed algorithm is performed efficiently in closed-form to avoid the meticulous choice of stepsizes, which is howe…
ESIEE（高等电子工程工程师学院）；Université Paris-Est（巴黎东大学）；CNRS（法国国家科学研究中心）；ENPC（国立桥路学校）；ESIEE Paris（巴黎高等电子工程工程师学院）；UPEM（巴黎东马恩拉瓦莱大学）；Facebook AI Research（Facebook人工智能研究院）；Université du Littoral Côte d’Opale（滨海奥帕尔海岸大学）
Space-filling designs such as scrambled-Hammersley, Latin Hypercube Sampling and Jittered Sampling have been proposed for fully parallel hyperparameter search, and were shown to be more effective than random or grid search. In this paper, we show that these designs only improve over random search by a constant factor. In contrast, we introduce a new approach based on reshaping the search distribution, which leads to substantial gains over random search, both theoretically and empirically. We propose two flavors of reshaping. First, when the distribution of the optimum is some known $P_0$, we p…
University of California, Los Angeles（加州大学洛杉矶分校）；Kwai Inc.（快手公司）；Tencent AI Lab（腾讯人工智能实验室）
Despite its groundbreaking success in Go and computer games, Monte Carlo Tree Search (MCTS) is computationally expensive as it requires a substantial number of rollouts to construct the search tree, which calls for effective parallelization. However, how to design effective parallel MCTS algorithms has not been systematically studied and remains poorly understood. In this paper, we seek to lay its first theoretical foundation, by examining the potential performance loss caused by parallelization when achieving a desired speedup. In particular, we discover the necessary conditions of achieving …
Amazon（亚马逊）；RIKEN AIP（理化学研究所先进智能项目中心）；Kyoto University（京都大学）；University of Oxford（牛津大学）
Neural architecture search (NAS) automates the design of deep neural networks. One of the main challenges in searching complex and non-continuous architectures is to compare the similarity of networks that the conventional Euclidean metric may fail to capture. Optimal transport (OT) is resilient to such complex structure by considering the minimal cost for transporting a network into another. However, the OT is generally not negative definite which may limit its ability to build the positive-definite kernels required in many kernel-dependent frameworks. Building upon tree-Wasserstein (TW), whi…
Korea Advanced Institute of Science and Technology（韩国高等科学技术研究院）
In this paper, a new population-guided parallel learning scheme is proposed to enhance the performance of off-policy reinforcement learning (RL). In the proposed scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information. The key point is that the information of the best policy is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search region by the multiple learners. The guidance by th…
École polytechnique（巴黎综合理工学院）；Argonne National Laboratory（阿贡国家实验室）；CNRS/Inria-LISN（法国国家科学研究中心/法国国家信息与自动化研究所-LISN）；U. Paris-Saclay（巴黎萨克雷大学）
Developing high-performing predictive models for large tabular data sets is a challenging task. The state-of-the-art methods are based on expert-developed model ensembles from different supervised learning methods. Recently, automated machine learning (AutoML) is emerging as a promising approach to automate predictive model development. Neural architecture search (NAS) is an AutoML approach that generates and evaluates multiple neural network architectures concurrently and improves the accuracy of the generated models iteratively. A key issue in NAS, particularly for large data sets, is the la…
Real-world problems often involve the optimization of several objectives under multiple constraints. An example is the hyper-parameter tuning problem of machine learning algorithms. In particular, the minimization of the estimation of the generalization error of a deep neural network and at the same time the minimization of its prediction time. We may also consider as a constraint that the deep neural network must be implemented in a chip with an area below some size. Here, both the objectives and the constraint are black boxes, i.e., functions whose analytical expressions are unknown and are …
