Previously, we proposed a probabilistic data generation model represented by an unobservable tree and a sequential updating method to calculate a posterior distribution over a set of trees. The set is called a meta-tree. In this paper, we propose a more efficient batch updating method.
Variants of Triplet networks are robust entities for learning a discriminative embedding subspace. There exist different triplet mining approaches for selecting the most suitable training triplets. Some of these mining methods rely on the extreme distances between instances, and some others make use of sampling. However, sampling from stochastic distributions of data rather than sampling merely from the existing embedding instances can provide more discriminative information. In this work, we sample triplets from distributions of data rather than from existing instances. We consider a multivar…
University of California at Santa Barbara（加州大学圣巴巴拉分校）
We develop adaptive replicated designs for Gaussian process metamodels of stochastic experiments. Adaptive batching is a natural extension of sequential design heuristics with the benefit of replication growing as response features are learned, inputs concentrate, and the metamodeling overhead rises. Motivated by the problem of learning the level set of the mean simulator response we develop four novel schemes: Multi-Level Batching (MLB), Ratchet Batching (RB), Adaptive Batched Stepwise Uncertainty Reduction (ABSUR), Adaptive Design with Stepwise Allocation (ADSA) and Deterministic Design with…
Finite element model updating utilizing frequency response functions as inputs is an important procedure in structural analysis, design and control. This paper presents a highly efficient framework that is built upon Gaussian process emulation to inversely identify model parameters through sampling. In particular, a multi-response Gaussian process (MRGP) meta-modeling approach is formulated that can accurately construct the error response surface, i.e., the discrepancies between the frequency response predictions and actual measurement. In order to reduce the computational cost of repeated fin…
Gaussian Processes (GPs) are powerful kernelized methods for non-parameteric regression used in many applications. However, their use is limited to a few thousand of training samples due to their cubic time complexity. In order to scale GPs to larger datasets, several sparse approximations based on so-called inducing points have been proposed in the literature. In this work we investigate the connection between a general class of sparse inducing point GP regression methods and Bayesian recursive estimation which enables Kalman Filter like updating for online learning. The majority of previous …
Hierarchical Bayesian models can be especially useful in precision medicine settings, where clinicians are interested in estimating the patient-level latent variables associated with an individual's current health state and its trajectory. Such models are often fit using batch Markov Chain Monte Carlo (MCMC). However, the slow speed of batch MCMC computation makes it difficult to implement in clinical settings, where immediate latent variable estimates are often desired in response to new patient data. In this report, we discuss how importance sampling (IS) can instead be used to obtain fast, …
University of Cambridge（剑桥大学）；Invenia Labs（Invenia实验室）；Microsoft Research（微软研究院）
Modern meta-learning approaches for image classification rely on increasingly deep networks to achieve state-of-the-art performance, making batch normalization an essential component of meta-learning pipelines. However, the hierarchical nature of the meta-learning setting presents several challenges that can render conventional batch normalization ineffective, giving rise to the need to rethink normalization in this setting. We evaluate a range of approaches to batch normalization for meta-learning scenarios, and develop a novel approach that we call TaskNorm. Experiments on fourteen datasets …
From drift to adaptation to the failed ml model: Transfer Learning in Industrial MLOps
University College London（伦敦大学学院）；The Alan Turing Institute（艾伦·图灵研究所）；Univeristy of Engineering and Technology（工程与技术大学）；Sahiwal Coal Fired Power Plant（萨希瓦尔燃煤发电厂）
Few-shot image classification (FSIC), which requires a model to recognize new categories via learning from few images of these categories, has attracted lots of attention. Recently, meta-learning based methods have been shown as a promising direction for FSIC. Commonly, they train a meta-learner (meta-learning model) to learn easy fine-tuning weight, and when solving an FSIC task, the meta-learner efficiently fine-tunes itself to a task-specific model by updating itself on few images of the task. In this paper, we propose a novel meta-learning based layer-wise adaptive updating (LWAU) method f…
Federated learning (FL) aims to train machine learning models in the decentralized system consisting of an enormous amount of smart edge devices. Federated averaging (FedAvg), the fundamental algorithm in FL settings, proposes on-device training and model aggregation to avoid the potential heavy communication costs and privacy concerns brought by transmitting raw data. However, through theoretical analysis we argue that 1) the multiple steps of local updating will result in gradient biases and 2) there is an inconsistency between the expected target distribution and the optimization objectives…
Adaptive Meta-Learning Stochastic Gradient Hamiltonian Monte Carlo Simulation for Bayesian Updating of Structural Dynamic Models
Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology, Harbin, China（工信部智能防灾减灾重点实验室，哈尔滨工业大学，哈尔滨，中国）；Key Lab of Structures Dynamic Behavior and Control of the Ministry of Education, Harbin Institute of Technology, Harbin, China（教育部结构动力行为与控制重点实验室，哈尔滨工业大学，哈尔滨，中国）；Division of Engineering and Applied Science, California Institute of Technology, CA, USA（加州理工学院工程与应用科学系，CA，美国）
Carnegie Mellon University（卡内基梅隆大学）；DEVCOM Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）
Supervised learning typically optimizes the expected value risk functional of the loss, but in many cases, we want to optimize for other risk functionals. In full-batch gradient descent, this is done by taking gradients of a risk functional of interest, such as the Conditional Value at Risk (CVaR) which ignores some quantile of extreme losses. However, deep learning must almost always use mini-batch gradient descent, and lack of unbiased estimators of various risk functionals make the right optimization procedure unclear. In this work, we introduce a meta-learning-based method of learning an i…
Most existing set encoding algorithms operate under the implicit assumption that all the set elements are accessible, and that there are ample computational and memory resources to load the set into memory during training and inference. However, both assumptions fail when the set is excessively large such that it is impossible to load all set elements into memory, or when data arrives in a stream. To tackle such practical challenges in large-scale set encoding, the general set-function constraints of permutation invariance and equivariance are not sufficient. We introduce a new property termed…
College of Biomedical Engineering & Instrument Science（生物医学工程与仪器科学学院）；Zhejiang University（浙江大学）
Data-free quantization is a task that compresses the neural network to low bit-width without access to original training data. Most existing data-free quantization methods cause severe performance degradation due to inaccurate activation clipping range and quantization error, especially for low bit-width. In this paper, we present a simple yet effective data-free quantization method with accurate activation clipping and adaptive batch normalization. Accurate activation clipping (AAC) improves the model accuracy by exploiting accurate activation information from the full-precision model. Adapti…
Texas A&M University（德克萨斯农工大学）；The University of Iowa（爱荷华大学）
Training of graph neural networks (GNNs) for large-scale node classification is challenging. A key difficulty lies in obtaining accurate hidden node representations while avoiding the neighborhood explosion problem. Here, we propose a new technique, named feature momentum (FM), that uses a momentum step to incorporate historical embeddings when updating feature representations. We develop two specific algorithms, known as GraphFM-IB and GraphFM-OB, that consider in-batch and out-of-batch data, respectively. GraphFM-IB applies FM to in-batch sampled data, while GraphFM-OB applies FM to out-of-b…
Indian Institute of Technology, Ropar（印度理工学院罗巴尔分校）
Meta-learning (ML) has emerged as a promising direction in learning models under constrained resource settings like few-shot learning. The popular approaches for ML either learn a generalizable initial model or a generic parametric optimizer through episodic training. The former approaches leverage the knowledge from a batch of tasks to learn an optimal prior. In this work, we study the importance of a batch for ML. Specifically, we first incorporate a batch episodic training regimen to improve the learning of the generic parametric optimizer. We also hypothesize that the common assumption in …
Meta-learning for sample-efficient Bayesian optimisation of fed-batch processes
Department of Computing, Imperial College London, United Kingdom（帝国理工学院伦敦分校计算机系）；BASF SE, Ludwigshafen, Germany（巴斯夫股份有限公司，德国路德维希沙芬）；School of Mathematical Sciences, Lancaster University, United Kingdom（兰卡斯特大学数学科学学院）；Department of Computer Science, University of Oxford, United Kingdom（牛津大学计算机科学系）
University of Washington（华盛顿大学）；Princeton University（普林斯顿大学）；Institute for Advanced Study（高等研究院）
A common challenge faced in practical supervised learning, such as medical image processing and robotic interactions, is that there are plenty of tasks but each task cannot afford to collect enough labeled examples to be learned in isolation. However, by exploiting the similarities across those tasks, one can hope to overcome such data scarcity. Under a canonical scenario where each task is drawn from a mixture of k linear regressions, we study a fundamental question: can abundant small-data tasks compensate for the lack of big-data tasks? Existing second moment based approaches show that such…
Batch Normalization (BN) and its variants have delivered tremendous success in combating the covariate shift induced by the training step of deep learning methods. While these techniques normalize feature distributions by standardizing with batch statistics, they do not correct the influence on features from extraneous variables or multiple distributions. Such extra variables, referred to as metadata here, may create bias or confounding effects (e.g., race when classifying gender from face images). We introduce the Metadata Normalization (MDN) layer, a new batch-level operation which can be us…
Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between w…
This is paper for the smooth function approximation by neural networks (NN). Mathematical or physical functions can be replaced by NN models through regression. In this study, we get NNs that generate highly accurate and highly smooth function, which only comprised of a few weight parameters, through discussing a few topics about regression. First, we reinterpret inside of NNs for regression; consequently, we propose a new activation function--integrated sigmoid linear unit (ISLU). Then special charateristics of metadata for regression, which is different from other data like image or sound, i…
In learning-to-learn the goal is to infer a learning algorithm that works well on a class of tasks sampled from an unknown meta distribution. In contrast to previous work on batch learning-to-learn, we consider a scenario where tasks are presented sequentially and the algorithm needs to adapt incrementally to improve its performance on future tasks. Key to this setting is for the algorithm to rapidly incorporate new observations into the model as they arrive, without keeping them in memory. We focus on the case where the underlying algorithm is ridge regression parameterized by a positive semi…
MetaWearS: A Shortcut in Wearable Systems Lifecycle with Only a Few Shots
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；University of Tehran（德黑兰大学）；Institute for Research in Fundamental Sciences (IPM)（基础科学研究所）；Lund University（隆德大学）
Batch effects can impair federated learning in multi-center omics studies
Institute for Computational Systems Biomedicine, University of Hamburg（汉堡大学计算系统生物医学研究所）；Chair of Proteomics and Bioanalytics, TUM School of Life Sciences, Technical University of Munich（慕尼黑技术大学生命科学学院蛋白质组学与生物分析系）；Department of Mathematics and Computer Science, University of Southern Denmark（丹麦南部大学数学与计算机科学系）；Biomedical Network Science Lab, Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡大学生物医学网络科学实验室）；Data Science in Systems Biology, TUM School of Life Sciences, Technical University of Munich（慕尼黑技术大学生命科学学院系统生物学数据科学）；Institute of Clinical Molecular Biology (IKMB), Kiel University and University Medical Center Schleswig-Holstein（基尔大学与石勒苏益格-荷尔斯泰因大学医学中心临床分子生物学研究所）
