Automated tuning of compute kernels is a popular area of research, mainly focused on finding optimal kernel parameters for a problem with fixed input sizes. This approach is good for deploying machine learning models, where the network topology is constant, but machine learning research often involves changing network topologies and hyperparameters. Traditional kernel auto-tuning has limited impact in this case; a more general selection of kernels is required for libraries to accelerate machine learning research.
In this paper we present initial results using machine learning to select kernels in a case study deploying high performance SYCL kernels in libraries that target a range of heterogeneous devices from desktop GPUs to embedded accelerators. The techniques investigated apply more generally and could similarly be integrated with other heterogeneous programming systems. By combining auto-tuning and machine learning these kernel selection processes can be deployed with little developer effort to achieve high performance on new hardware.
Automatically tuning parallel compute kernels allows libraries and frameworks to achieve performance on a wide range of hardware, however these techniques are typically focused on finding optimal kernel parameters for particular input sizes and parameters. General purpose compute libraries must be able to cater to all inputs and parameters provided by a user, and so these techniques are of limited use. Additionally, parallel programming frameworks such as SYCL require that the kernels be deployed in a binary format embedded within the library. As such it is impractical to deploy a large number…
Over recent years heterogeneous systems have become more prevalent across HPC systems, with over 100 supercomputers in the TOP500 incorporating GPUs or other accelerators. These hardware platforms have different performance characteristics and optimization requirements. In order to make the most of multiple accelerators a developer has to provide implementations of their algorithms tuned for each device. Hardware vendors provide libraries targeting their devices specifically, which provide good performance but frequently have different API designs, hampering portability. The SYCL programming m…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN infere…
University of Michigan（密歇根大学）；Columbia University（哥伦比亚大学）
Evaluation and validation of complicated control systems are crucial to guarantee usability and safety. Usually, failure happens in some very rarely encountered situations, but once triggered, the consequence is disastrous. Accelerated Evaluation is a methodology that efficiently tests those rarely-occurring yet critical failures via smartly-sampled test cases. The distribution used in sampling is pivotal to the performance of the method, but building a suitable distribution requires case-by-case analysis. This paper proposes a versatile approach for constructing sampling distribution using ke…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Alan Turing Institute（艾伦·图灵研究所）
Clinical prognostic models derived from largescale healthcare data can inform critical diagnostic and therapeutic decisions. To enable off-theshelf usage of machine learning (ML) in prognostic research, we developed AUTOPROGNOSIS: a system for automating the design of predictive modeling pipelines tailored for clinical prognosis. AUTOPROGNOSIS optimizes ensembles of pipeline configurations efficiently using a novel batched Bayesian optimization (BO) algorithm that learns a low-dimensional decomposition of the pipelines high-dimensional hyperparameter space in concurrence with the BO procedure.…
BOOST: A Data-Driven Framework for the Automated Joint Selection of Kernel and Acquisition Functions in Bayesian Optimization
Department of Chemical and Biomolecular Engineering, Korea Advanced Institute of Science and Technology（化学与生物分子工程系，韩国科学技术院）；Department of AX, Korea Advanced Institute of Science and Technology（AX系，韩国科学技术院）；Saudi Aramco-KAIST CO2 Management Center（沙特阿美-KAIST二氧化碳管理中心）
Towards Automated Kernel Generation in the Era of LLMs
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Normal University（北京师范大学）；Peking University（北京大学）；Beijing Institute of Technology（北京理工大学）；Cornell University（康奈尔大学）；Beijing Jiaotong University（北京交通大学）；Renmin University of China（中国人民大学）；Hong Kong University of Science and Technology (Guangzhou)（广州科技大学）
We consider the two-group classification problem and propose a kernel classifier based on the optimal scoring framework. Unlike previous approaches, we provide theoretical guarantees on the expected risk consistency of the method. We also allow for feature selection by imposing structured sparsity using weighted kernels. We propose fully-automated methods for selection of all tuning parameters, and in particular adapt kernel shrinkage ideas for ridge parameter selection. Numerical studies demonstrate the superior classification performance of the proposed approach compared to existing nonparam…
University of Oxford（牛津大学）；Universitat de València（瓦伦西亚大学）
Current adoption of machine learning in industrial, societal and economical activities has raised concerns about the fairness, equity and ethics of automated decisions. Predictive models are often developed using biased datasets and thus retain or even exacerbate biases in their decisions and recommendations. Removing the sensitive covariates, such as gender or race, is insufficient to remedy this issue since the biases may be retained due to other related covariates. We present a regularization approach to this problem that trades off predictive accuracy of the learned models (with respect to…
Domain Switching on the Pareto Front: Multi-Objective Deep Kernel Learning in Automated Piezoresponse Force Microscopy
University of Tennessee（田纳西大学）；Institute of Science Tokyo（东京科学大学）；University of California, Berkeley（加州大学伯克利分校）；Rice University（莱斯大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）
Bosch Center for Artificial Intelligence（博世人工智能中心）
Despite recent advances in automated machine learning, model selection is still a complex and computationally intensive process. For Gaussian processes (GPs), selecting the kernel is a crucial task, often done manually by the expert. Additionally, evaluating the model selection criteria for Gaussian processes typically scales cubically in the sample size, rendering kernel search particularly computationally expensive. We propose a novel, efficient search method through a general, structured kernel space. Previous methods solved this task via Bayesian optimization and relied on measuring the di…
Università degli Studi di Genova（热那亚大学）；Massachusetts Institute of Technology（麻省理工学院）；Istituto Italiano di Tecnologia（意大利理工学院）
Kernel methods provide a principled approach to nonparametric learning. While their basic implementations scale poorly to large problems, recent advances showed that approximate solvers can efficiently handle massive datasets. A shortcoming of these solutions is that hyperparameter tuning is not taken care of, and left for the user to perform. Hyperparameters are crucial in practice and the lack of automated tuning greatly hinders efficiency and usability. In this paper, we work to fill in this gap focusing on kernel ridge regression based on the Nyström approximation. After reviewing and cont…
Active learning methods are rapidly becoming the integral component of automated experiment workflows in imaging, materials synthesis, and computation. The distinctive aspect of many experimental scenarios is the presence of multiple information channels, including both the intrinsic modalities of the measurement system and the exogenous environment and noise signals. One of the key tasks in experimental studies is hence establishing which of these channels is predictive of the behaviors of interest. Here we explore the problem of discovery of the optimal predictive channel for structure-prope…
New University of Lisbon（新里斯本大学）；Radboud University Nijmegen（拉德堡德大学奈梅亨分校）
Smart premise selection is essential when using automated reasoning as a tool for large-theory formal proof development. A good method for premise selection in complex mathematical libraries is the application of machine learning to large corpora of proofs. This work develops learning-based premise selection in two ways. First, a newly available minimal dependency analysis of existing high-level formal mathematical proofs is used to build a large knowledge base of proof dependencies, providing precise data for ATP-based re-verification and for training premise selection algorithms. Second, a n…
University of Cambridge（剑桥大学）；UCLA（加州大学洛杉矶分校）；The Alan Turing Institute（艾伦·图灵研究所）
An essential problem in automated machine learning (AutoML) is that of model selection. A unique challenge in the sequential setting is the fact that the optimal model itself may vary over time, depending on the distribution of features and labels available up to each point in time. In this paper, we propose a novel Bayesian optimization (BO) algorithm to tackle the challenge of model selection in this setting. This is accomplished by treating the performance at each time step as its own black-box function. In order to solve the resulting multiple black-box function optimization problem jointl…
HieraEdgeNet: A Multi-Scale Edge-Enhanced Framework for Automated Pollen Recognition
School of Agriculture and Biology, Shanghai Jiao Tong University（上海交通大学农业生物学学院）；Shanghai Forestry Station（上海市林业站）；Shanghai Yangtze River Delta Eco-Environmental Change and Management Observation and Research Station (Shanghai Urban Ecosystem Research Station)（上海长江三角洲生态环境变化与管理观测与研究站（上海城市生态系统研究站））
Netherlands eScience Center（荷兰电子科学中心）；University of Amsterdam（阿姆斯特丹大学）
Finding optimal parameter configurations for tunable GPU kernels is a non-trivial exercise for large search spaces, even when automated. This poses an optimization task on a non-convex search space, using an expensive to evaluate function with unknown derivative. These characteristics make a good candidate for Bayesian Optimization, which has not been applied to this problem before. However, the application of Bayesian Optimization to this problem is challenging. We demonstrate how to deal with the rough, discrete, constrained search spaces, containing invalid configurations. We introduce a no…
Università di Firenze（佛罗伦萨大学）；INFN（意大利国家核物理研究所）；CSDC（复杂系统动力学研究中心）；University of Namur（那慕尔大学）；University of Lisbon（里斯本大学）；Instituto de Telecomunicacoes（电信研究所）
A novel strategy to automated classification is introduced which exploits a fully trained dynamical system to steer items belonging to different categories toward distinct asymptotic attractors. These latter are incorporated into the model by taking advantage of the spectral decomposition of the operator that rules the linear evolution across the processing network. Non-linear terms act for a transient and allow to disentangle the data supplied as initial condition to the discrete dynamical system, shaping the boundaries of different attractors. The network can be equipped with several memory …
We analyze a prospect for predicting gravitational waveforms from compact binaries based on automated machine learning (AutoML) from around a hundred different possible regression models, without having to resort to tedious and manual case-by-case analyses and fine-tuning. The particular study of this article is within the context of the gravitational waves emitted by the collision of two spinless black holes in initial quasi-circular orbit. We find, for example, that approaches such as Gaussian process regression with radial bases as kernels do provide a sufficiently accurate solution, an app…
