Machine learning methods adapt the parameters of a model, constrained to lie in a given model class, by using a fixed learning procedure based on data or active observations. Adaptation is done on a per-task basis, and retraining is needed when the system configuration changes. The resulting inefficiency in terms of data and training time requirements can be mitigated, if domain knowledge is available, by selecting a suitable model class and learning procedure, collectively known as inductive bias. However, it is generally difficult to encode prior knowledge into an inductive bias, particularly with black-box model classes such as neural networks. Meta-learning provides a way to automatize the selection of an inductive bias. Meta-learning leverages data or active observations from tasks that are expected to be related to future, and a priori unknown, tasks of interest. With a meta-trained inductive bias, training of a machine learning model can be potentially carried out with reduced training data and/or time complexity. This paper provides a high-level introduction to meta-learning with applications to communication systems.
Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation
Department of Electrical and Computer Engineering（电气与计算机工程系）；University of Minnesota（明尼苏达大学）；Department of Computer Science（计算机科学系）；ETH Zürich（苏黎世联邦理工学院）
Meta-learners and ensembles aim to combine a set of relevant yet diverse base models to improve predictive performance. However, determining an appropriate set of base models is challenging, especially in online environments where the underlying distribution of data can change over time. In this paper, we present a novel approach for estimating the conceptual similarity of base models, which is calculated using the Principal Angles (PAs) between their underlying subspaces. We propose two methods that use conceptual similarity as a metric to obtain a relevant yet diverse subset of base models: …
Manifold meta-learning for reduced-complexity neural system identification
SUPSI（瑞士应用科学大学）；IDSIA - Dalle Molle Institute for Artificial Intelligence（达勒·莫勒人工智能研究所）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机美国研究实验室）；IMT School for Advanced Studies Lucca（卢卡高等研究院）
Convolution is one of the fundamental operations of deep neural networks with demanding matrix computation. In a graphic processing unit (GPU), Tensor Core is a specialized matrix processing hardware equipped with reduced-precision matrix-multiply-accumulate (MMA) instructions to increase throughput. However, it is challenging to achieve optimal performance since the best scheduling of MMA instructions varies for different convolution sizes. In particular, reduced-precision MMA requires many elements grouped as a matrix operand, seriously limiting data reuse and imposing packing and layout ove…
Adaptive Pruning for Increased Robustness and Reduced Computational Overhead in Gaussian Process Accelerated Saddle Point Searches
Rohit Goswami Institute IMX and Lab-COSMO（Rohit Goswami 实验室IMX和Lab-COSMO）；École polytechnique fédérale de Lausanne (EPFL)（联邦理工学院洛桑校区）；Science Institute & Faculty of Physical Sciences University of Iceland（冰岛大学科学研究院及物理科学学院）
Quantum feature-map learning with reduced resource overhead
Department of Computer Science and Institute of Applied Mathematics, University of British Columbia (UBC), Vancouver, B.C. V6T 1Z4, Canada（计算机科学系和应用数学研究所，不列颠哥伦比亚大学（UBC），温哥华，B.C. V6T 1Z4，加拿大）；Stewart Blusson Quantum Matter Institute (QMI), Vancouver, B.C. V6T 1Z4, Canada（斯图尔特·布卢森量子物质研究所（QMI），温哥华，B.C. V6T 1Z4，加拿大）；Institute of Physics, University of Freiburg, Freiburg (Breisgau), 79104, Germany（物理研究所，弗赖堡大学，弗赖堡（巴登-符腾堡），79104，德国）；Department of Chemistry, University of British Columbia (UBC), Vancouver, B.C. V6T 1Z1, Canada（化学系，不列颠哥伦比亚大学（UBC），温哥华，B.C. V6T 1Z1，加拿大）
Split learning is a promising privacy-preserving distributed learning scheme that has low computation requirement at the edge device but has the disadvantage of high communication overhead between edge device and server. To reduce the communication overhead, this paper proposes a loss-based asynchronous training scheme that updates the client-side model less frequently and only sends/receives activations/gradients in selected epochs. To further reduce the communication overhead, the activations/gradients are quantized using 8-bit floating point prior to transmission. An added benefit of the pr…
In this paper, we propose Code-Bridged Classifier (CBC), a framework for making a Convolutional Neural Network (CNNs) robust against adversarial attacks without increasing or even by decreasing the overall models' computational complexity. More specifically, we propose a stacked encoder-convolutional model, in which the input image is first encoded by the encoder module of a denoising auto-encoder, and then the resulting latent representation (without being decoded) is fed to a reduced complexity CNN for image classification. We illustrate that this network not only is more robust to adversari…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon Web Services（亚马逊云科技）
When scaling distributed training, the communication overhead is often the bottleneck. In this paper, we propose a novel SGD variant with reduced communication and adaptive learning rates. We prove the convergence of the proposed algorithm for smooth but non-convex problems. Empirical results show that the proposed algorithm significantly reduces the communication overhead, which, in turn, reduces the training time by up to 30% for the 1B word dataset.
University of Connecticut（康涅狄格大学）；Northeastern University（东北大学）；Lehigh University（理海大学）；Duke University（杜克大学）；University of Rochester（罗切斯特大学）
The proliferation of deep learning (DL) has led to the emergence of privacy and security concerns. To address these issues, secure Two-party computation (2PC) has been proposed as a means of enabling privacy-preserving DL computation. However, in practice, 2PC methods often incur high computation and communication overhead, which can impede their use in large-scale systems. To address this challenge, we introduce RRNet, a systematic framework that aims to jointly reduce the overhead of MPC comparison protocols and accelerate computation through hardware acceleration. Our approach integrates th…
Federated learning (FL) is able to manage edge devices to cooperatively train a model while maintaining the training data local and private. One common assumption in FL is that all edge devices share the same machine learning model in training, for example, identical neural network architecture. However, the computation and store capability of different devices may not be the same. Moreover, reducing communication overheads can improve the training efficiency though it is still a challenging problem in FL. In this paper, we propose a novel FL method, called FedHe, inspired by knowledge distill…
Sensors with digital outputs require software conversion routines to transform the unitless analogue-to-digital converter samples to physical quantities with correct units. These conversion routines are computationally complex given the limited computational resources of low-power embedded systems. This article presents a set of machine learning methods to learn new, less-complex conversion routines that do not sacrifice accuracy for the BME680 environmental sensor. We present a Pareto analysis of the tradeoff between accuracy and computational overhead for the models and models that reduce th…
