近年来，大量数据不断产生，计算能力也持续增长。这一形势促使人们对人工神经网络的兴趣重新兴起。训练有效神经网络模型的主要挑战之一，是找到合适的超参数组合。事实上，选择适当的超参数空间搜索方法会直接影响所得神经网络模型的准确率。常见的超参数优化方法包括 Grid Search、Random Search 和 Bayesian Optimization，此外还有 CMA-ES 等基于种群的方法。本文提出 HBRKGA，这是一种用于超参数优化的新型基于种群的方法。HBRKGA 是一种混合方法，将 Biased Random Key Genetic Algorithm（偏置随机密钥遗传算法）与 Random Walk 技术相结合，以高效搜索超参数空间。为评估所提方法的有效性，研究在八个不同数据集上进行了多项计算实验。结果表明，HBRKGA 能够找到的超参数配置在八个数据集中的六个上（在预测质量方面）优于基线方法，同时表现出合理的执行时间。
英文摘要：
In recent years, large amounts of data have been generated, and computer power has kept growing. This scenario has led to a resurgence in the interest in artificial neural networks. One of the main challenges in training effective neural network models is finding the right combination of hyperparameters to be used. Indeed, the choice of an adequate approach to search the hyperparameter space directly influences the accuracy of the resulting neural network model. Common approaches for hyperparameter optimization are Grid Search, Random Search, and Bayesian Optimization. There are also population-based methods such as CMA-ES. In this paper, we present HBRKGA, a new population-based approach for hyperparameter optimization. HBRKGA is a hybrid approach that combines the Biased Random Key Genetic Algorithm with a Random Walk technique to search the hyperparameter space efficiently. Several computational experiments on eight different datasets were performed to assess the effectiveness of the proposed approach. Results showed that HBRKGA could find hyperparameter configurations that outperformed (in terms of predictive quality) the baseline methods in six out of eight datasets while showing a reasonable execution time.
We propose a genetic algorithm (GA) for hyperparameter optimization of artificial neural networks which includes chromosomal crossover as well as a decoupling of parameters (i.e., weights and biases) from hyperparameters (e.g., learning rate, weight decay, and dropout) during sexual reproduction. Children are produced from three parents; two contributing hyperparameters and one contributing the parameters. Our version of population-based training (PBT) combines traditional gradient-based approaches such as stochastic gradient descent (SGD) with our GA to optimize both parameters and hyperparam…
Knight Foundation School of Computing and Information Sciences, Florida International University（佛罗里达国际大学奈特基金会计算与信息科学学院）；Sustainability, Optimization, and Learning for InterDependent networks (solid) laboratory, FIU（佛罗里达国际大学可持续性、优化与相互依赖网络学习实验室）；Department of Computer Science, University of Georgia（佐治亚大学计算机科学系）
Hyperparameter tuning in machine learning algorithms is a computationally challenging task due to the large-scale nature of the problem. In order to develop an efficient strategy for hyper-parameter tuning, one promising solution is to use swarm intelligence algorithms. Artificial Bee Colony (ABC) optimization lends itself as a promising and efficient optimization algorithm for this purpose. However, in some cases, ABC can suffer from a slow convergence rate or execution time due to the poor initial population of solutions and expensive objective functions. To address these concerns, a novel a…
Despite a series of recent successes in reinforcement learning (RL), many RL algorithms remain sensitive to hyperparameters. As such, there has recently been interest in the field of AutoRL, which seeks to automate design decisions to create more general algorithms. Recent work suggests that population based approaches may be effective AutoRL algorithms, by learning hyperparameter schedules on the fly. In particular, the PB2 algorithm is able to achieve strong performance in RL tasks by formulating online hyperparameter optimization as time varying GP-bandit problem, while also providing theor…
Hyperparameter Optimization via Interacting with Probabilistic Circuits
Computer Science Department, Technical University Darmstadt（技术大学达姆施塔特计算机科学系）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；German Research Center for AI (DFKI)（德国人工智能研究中心）
Many of the recent triumphs in machine learning are dependent on well-tuned hyperparameters. This is particularly prominent in reinforcement learning (RL) where a small change in the configuration can lead to failure. Despite the importance of tuning hyperparameters, it remains expensive and is often done in a naive and laborious way. A recent solution to this problem is Population Based Training (PBT) which updates both weights and hyperparameters in a single training run of a population of agents. PBT has been shown to be particularly effective in RL, leading to widespread use in the field. …
Data augmentation is an effective technique to improve the generalization of deep neural networks. Recently, AutoAugment proposed a well-designed search space and a search algorithm that automatically finds augmentation policies in a data-driven manner. However, AutoAugment is computationally intensive. In this paper, we propose an efficient gradient-based search algorithm, called Hypernetwork-Based Augmentation (HBA), which simultaneously learns model parameters and augmentation hyperparameters in a single training. Our HBA uses a hypernetwork to approximate a population-based training algori…
Microsoft（微软）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Berkeley（加州大学伯克利分校）；International Computer Science Institute（国际计算机科学研究所）
Pruning is an effective method to reduce the memory footprint and computational cost associated with large natural language processing models. However, current pruning algorithms either only focus on one pruning category, e.g., structured pruning and unstructured, or need extensive hyperparameter tuning in order to get reasonable accuracy performance. To address these challenges, we propose LEArnable Pruning (LEAP), an effective method to gradually prune the model based on thresholds learned by gradient descent. Different than previous learnable pruning methods, which utilize $L_0$ or $L_1$ pe…
Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisation algorithm. In this work we present \emph{Population Based Training (PBT)}, a simple asynchronous optimisation algorithm which effectively utilises a fixed computational budget to jointly optimise a population of models and their hyperparameters to maximise performance. Importantly, PBT discovers a schedule of hyperparameter settings rather than following the generally sub-…
Time-Efficient Hybrid Hyperparameter Tuning Approach for Cardiovascular Disease Classification
Department of Computer Science, Institute of Science, Banaras Hindu University（计算机科学系，科学学院，班纳拉森胡大学）；School of Computer Science, University of Petroleum and Energy Studies（计算机科学学院，石油与能源研究大学）
We demonstrate the ability of hybrid regularization methods to automatically avoid the double descent phenomenon arising in the training of random feature models (RFM). The hallmark feature of the double descent phenomenon is a spike in the regularization gap at the interpolation threshold, i.e. when the number of features in the RFM equals the number of training samples. To close this gap, the hybrid method considered in our paper combines the respective strengths of the two most common forms of regularization: early stopping and weight decay. The scheme does not require hyperparameter tuning…
Two-Time-Scale Learning Dynamics: A Population View of Neural Network Training
Maxwell Institute for Mathematical Sciences and School of Mathematical and Computer Sciences Heriot–Watt University, Edinburgh, UK（麦克斯韦数学科学研究所和赫里奥特-沃特大学数学与计算机科学学院，爱丁堡，英国）；Department of Mathematics and Computer Science University of Ferrara, Italy（费拉拉大学数学与计算机科学系）
Reinforcement-Guided Hyper-Heuristic Hyperparameter Optimization for Fair and Explainable Spiking Neural Network-Based Financial Fraud Detection
organization= Department of Computer Science \& Engineering, Khulna University of Engineering \& Technology (KUET) , city= Khulna , postcode= 9203 , country= Bangladesh；organization= Thomas Lord Department of Computer Science, Viterbi School of Engineering, University of Southern California , city= Los Angeles , state= CA , postcode= 90089 , country= USA；organization= Department of Computer Science, American International University-Bangladesh (AIUB) , city= Dhaka , postcode= 1229 , country= Bangladesh
Reinforcement learning (RL) offers the potential for training generally capable agents that can interact autonomously in the real world. However, one key limitation is the brittleness of RL algorithms to core hyperparameters and network architecture choice. Furthermore, non-stationarities such as evolving training data and increased agent complexity mean that different hyperparameters and architectures may be optimal at different points of training. This motivates AutoRL, a class of methods seeking to automate these design choices. One prominent class of AutoRL methods is Population-Based Trai…
Population Based Training (PBT) is a recent approach that jointly optimizes neural network weights and hyperparameters which periodically copies weights of the best performers and mutates hyperparameters during training. Previous PBT implementations have been synchronized glass-box systems. We propose a general, black-box PBT framework that distributes many asynchronous "trials" (a small number of training steps with warm-starting) across a cluster, coordinated by the PBT controller. The black-box design does not make assumptions on model architectures, loss functions or training procedures. O…
National Chiao Tung University（国立交通大学）；University of Alberta（阿尔伯塔大学）；Pervasive Artificial Intelligence Research (PAIR) Labs（普适人工智能研究实验室）
AlphaZero has been very successful in many games. Unfortunately, it still consumes a huge amount of computing resources, the majority of which is spent in self-play. Hyperparameter tuning exacerbates the training cost since each hyperparameter configuration requires its own time to train one run, during which it will generate its own self-play records. As a result, multiple runs are usually needed for different hyperparameter configurations. This paper proposes using population based training (PBT) to help tune hyperparameters dynamically and improve strength during training time. Another sign…
University of Freiburg（弗莱堡大学）；German Cancer Research Center（德国癌症研究中心）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters. This sensitivity can partly be attributed to the non-stationarity of the RL problem, potentially requiring different hyperparameter settings at various stages of the learning process. Additionally, in the RL setting, hyperparameter optimization (HPO) requires a large number of environment interactions, hindering the transfer of the successes in RL to real-world applicatio…
A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning
Brij Disa Centre for Data Science and Artificial Intelligence（布里吉·迪萨数据科学与人工智能中心）；Indian Institute of Management Ahmedabad（印度管理学院艾哈迈达巴德分校）；Department of Statistics and Data Science（统计与数据科学系）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）
University of Georgia（佐治亚大学）；Santa Clara University（圣克拉拉大学）；Alibaba Group（阿里巴巴集团）
Owing to the remarkable capability of extracting effective graph embeddings, graph convolutional network (GCN) and its variants have been successfully applied to a broad range of tasks, such as node classification, link prediction, and graph classification. Traditional GCN models suffer from the issues of overfitting and oversmoothing, while some recent techniques like DropEdge could alleviate these issues and thus enable the development of deep GCN. However, training GCN models is non-trivial, as it is sensitive to the choice of hyperparameters such as dropout rate and learning weight decay, …
Cognizant AI Labs（高知特人工智能实验室）；The Univ. of Texas at Austin（德克萨斯大学奥斯汀分校）
Metalearning of deep neural network (DNN) architectures and hyperparameters has become an increasingly important area of research. At the same time, network regularization has been recognized as a crucial dimension to effective training of DNNs. However, the role of metalearning in establishing effective regularization has not yet been fully explored. There is recent evidence that loss-function optimization could play this role, however it is computationally impractical as an outer loop to full training. This paper presents an algorithm called Evolutionary Population-Based Training (EPBT) that…
An LSTM-PINN Hybrid Method to the specific problem of population forecasting
Nanophotonics and Biophotonics Key Laboratory of Jilin Province, School of Physics, Changchun University of Science and Technology（吉林省纳米光子与生物光子重点实验室，长春理工大学物理学院）
Multi-objective optimization and quantum hybridization of equivariant deep learning interatomic potentials
Terra Quantum AG；LIACS, Leiden University（LIACS，莱顿大学）；Nanoscience Center and Department of Chemistry, University of Jyväskylä（贾瓦尔基利亚大学纳米科学中心和化学系）；Department of Materials Engineering, Indian Institute of Science（印度科学研究所材料工程系）；Schaeffler Technologies AG & Co. KG
ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learning
Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）
Continuing advances in neural interfaces have enabled simultaneous monitoring of spiking activity from hundreds to thousands of neurons. To interpret these large-scale data, several methods have been proposed to infer latent dynamic structure from high-dimensional datasets. One recent line of work uses recurrent neural networks in a sequential autoencoder (SAE) framework to uncover dynamics. SAEs are an appealing option for modeling nonlinear dynamical systems, and enable a precise link between neural activity and behavior on a single-trial basis. However, the very large parameter count and co…
Forecasting India's Demographic Transition Under Fertility Policy Scenarios Using hybrid LSTM-PINN Model
Sant Longowal Institute of Engineering and Technology（桑特·朗戈瓦尔工程与技术学院）；School of Computer and Mathematical Sciences, The University of Adelaide（计算机与数学科学学院，阿德莱德大学）
Hybrid quantum tensor networks for aeroelastic applications
Institute for AI Safety and Security（人工智能安全与保安研究所）；German Aerospace Center (DLR)（德国航空航天中心）；Institute of Aeroelasticity（气动弹性研究所）；Institute of Neural Information Processing（神经信息处理研究所）；University of Ulm（乌尔姆大学）
Hybrid Adaptive Conformal Offline Reinforcement Learning for Fair Population Health Management
Waymark, San Francisco, CA, USA（Waymark）；San Francisco General Hospital, University of California San Francisco, San Francisco, CA, USA（旧金山通用医院，加州大学旧金山分校）；University of California San Francisco, San Francisco, CA, USA（加州大学旧金山分校）；University of Pennsylvania, Philadelphia, PA, USA（宾夕法尼亚大学）
Wuhan University（武汉大学）；The University of Sydney（悉尼大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；the University of Tokyo（东京大学）
In neural networks, developing regularization algorithms to settle overfitting is one of the major study areas. We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs), including drop rates and weight matrices, has been developed based on the proposed upper bound of the local Rademacher complexity by the strict mathematical deduction. The analyses of dropout in FCNs and DropBlock in CNNs with keep rate matrices in diffe…
Convolutional Neural Networks (CNNs) have demonstrated their superiority in image classification, and evolutionary computation (EC) methods have recently been surging to automatically design the architectures of CNNs to save the tedious work of manually designing CNNs. In this paper, a new hybrid differential evolution (DE) algorithm with a newly added crossover operator is proposed to evolve the architectures of CNNs of any lengths, which is named DECNN. There are three new ideas in the proposed DECNN method. Firstly, an existing effective encoding scheme is refined to cater for variable-leng…
Accelerating physics-informed neural networks for full waveform inversion using a hybrid quantum-classical finite-basis architecture
Department of Geophysics, Colorado School of Mines（地质学系，科罗拉多矿业学院）；Department of Energy Science and Engineering, Stanford University（能源科学与工程系，斯坦福大学）；Department of Petroleum Engineering, Colorado School of Mines（石油工程系，科罗拉多矿业学院）
