Artificial Neural Networks (ANNs) are known as state-of-the-art techniques in Machine Learning (ML) and have achieved outstanding results in data-intensive applications, such as recognition, classification, and segmentation. These networks mostly use deep layers of convolution or fully connected layers with many filters in each layer, demanding a large amount of data and tunable hyperparameters to achieve competitive accuracy. As a result, storage, communication, and computational costs of training (in particular training time) become limiting factors to scale them up. In this paper, we propose a new training methodology for ANNs that exploits the observation of improvement of accuracy shows temporal variations which allow us to skip updating weights when the variation is minuscule. During such time windows, we keep updating bias which ensures the network still trains and avoids overfitting; however, we selectively skip updating weights (and their time-consuming computations). Such a training approach virtually achieves the same accuracy with considerably less computational cost, thus lower training time. We propose two methods for updating weights and evaluate them by analyzing four state-of-the-art models, AlexNet, VGG-11, VGG-16, ResNet-18 on CIFAR datasets. On average, our two proposed methods called WUS and WUS+LR reduced the training time (compared to the baseline) by 54%, and 50%, respectively on CIFAR-10; and 43% and 35% on CIFAR-100, respectively.
Université de Caen（卡昂大学）；ENSICAEN（卡昂国立高等工程师学院）；CNRS（法国国家科学研究中心）；Institut Mines-Telecom sudParis（巴黎矿业电信学院南巴黎分校）
Semi-supervised template update systems allow to automatically take into account the intra-class variability of the biometric data over time. Such systems can be inefficient by including too many impostor's samples or skipping too many genuine's samples. In the first case, the biometric reference drifts from the real biometric data and attracts more often impostors. In the second case, the biometric reference does not evolve quickly enough and also progressively drifts from the real biometric data. We propose a hybrid system using several biometric sub-references in order to increase per- form…
I present my recollections of Richard Feynman's mid-1980s interest in artificial intelligence and neural networks, set in the technical context of the physics-related approaches to neural networks of that time. I attempt to evaluate his ideas in the light of the substantial advances in the field since then, and vice versa. There are aspects of Feynman's interests that I think have been largely achieved and others that remain excitingly open, notably in computational science, and potentially including the revival of symbolic methods therein.
A Federated Artificial Intelligence Framework for Optimizing Pancreatic Cancer Treatment - Strategy Update
University Medical Center Göttingen（哥廷根大学医学中心）；University of Göttingen（哥廷根大学）；Justus-Liebig University（吉森尤斯图斯-李比希大学）；Technical University of Munich（慕尼黑工业大学）；TUM School of Medicine and Health（慕尼黑工业大学医学与健康学院）；Technical University of Denmark（丹麦技术大学）；Max Planck Institute for Biology of Ageing（马克斯·普朗克衰老生物学研究所）；Philipps-University Marburg（马尔堡菲利普斯大学）
European Southern Observatory (ESO)（欧洲南方天文台）；Boston University（波士顿大学）；Joint ALMA Observatory（阿塔卡马大型毫米波/亚毫米波阵列联合天文台）；UNED（西班牙国家远程教育大学）
We model the spectral energy distributions (SEDs) of 23 protoplanetary disks in the Taurus-Auriga star-forming region using detailed disk models and a Bayesian approach. This is made possible by combining these models with artificial neural networks to drastically speed up their performance. Such a setup allows us to confront $α$-disk models with observations while accounting for several uncertainties and degeneracies. Our results yield high viscosities and accretion rates for many sources, which is not consistent with recent measurements of low turbulence levels in disks. This inconsistency c…
Artificial Entanglement in the Fine-Tuning of Large Language Models
Department of Computer Science, University of Pittsburgh, Pittsburgh, PA 15260, USA（匹兹堡大学计算机科学系）；Department of Computer Science, Northwestern University, Evanston, IL 60208, USA（西北大学计算机科学系）
Department of Electrical, Electronic and Computer Engineering, University of Ulsan（电气电子与计算机工程系，釜山大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（电气与计算机工程系，全北大学）；Department of Electrical and Computer Engineering, INMC, Seoul National University（电气与计算机工程系，首尔国立大学）
Institute for Information Systems, FHNW University of Applied Sciences and Arts Northwestern Switzerland（瑞士西北应用科学与艺术大学信息系统研究所）；Chair for Philosophy, ETH Zurich（苏黎世联邦理工学院哲学教席）
Many computer models such as cellular automata and artificial neural networks have been developed and successfully applied. However, in some cases, these models might be restrictive on the possible solutions or their solutions might be difficult to interpret. To overcome this problem, we outline a new approach, the so-called allagmatic method, that automatically programs and executes models with as little limitations as possible while maintaining human interpretability. Earlier we described a metamodel and its building blocks according to the philosophical concepts of structure (spatial dimens…
Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam -- a multiplicative version of the Adam optimiser -- and show that it can train state of the art neural network architectures without learning rate tuning.…
Texas A&M University（德州农工大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
Training of deep neural networks (DNNs) frequently involves optimizing several millions or even billions of parameters. Even with modern computing architectures, the computational expense of DNN training can inhibit, for instance, network architecture design optimization, hyper-parameter studies, and integration into scientific research cycles. The key factor limiting performance is that both the feed-forward evaluation and the back-propagation rule are needed for each weight during optimization in the update rule. In this work, we propose a method to decouple the evaluation of the update rule…
Slumbering to Precision: Enhancing Artificial Neural Network Calibration Through Sleep-like Processes
Department of Computer Science \& Engineering, University of California, San Diego, La Jolla, California, USA；ARTISAN, Georgia Institute of Technology, Atlanta, Georgia, USA；Department of Medicine, University of California, San Diego, La Jolla, California, USA
The brain, as the source of inspiration for Artificial Neural Networks (ANN), is based on a sparse structure. This sparse structure helps the brain to consume less energy, learn easier and generalize patterns better than any other ANN. In this paper, two evolutionary methods for adopting sparsity to ANNs are proposed. In the proposed methods, the sparse structure of a network as well as the values of its parameters are trained and updated during the learning process. The simulation results show that these two methods have better accuracy and faster convergence while they need fewer training sa…
Free University of Berlin（柏林自由大学）；RUDN University（俄罗斯人民友谊大学）
The paper deals with learning probability distributions of observed data by artificial neural networks. We suggest a so-called gradient conjugate prior (GCP) update appropriate for neural networks, which is a modification of the classical Bayesian update for conjugate priors. We establish a connection between the gradient conjugate prior update and the maximization of the log-likelihood of the predictive distribution. Unlike for the Bayesian neural networks, we use deterministic weights of neural networks, but rather assume that the ground truth distribution is normal with unknown mean and var…
Modern speech enhancement algorithms achieve remarkable noise suppression by means of large recurrent neural networks (RNNs). However, large RNNs limit practical deployment in hearing aid hardware (HW) form-factors, which are battery powered and run on resource-constrained microcontroller units (MCUs) with limited memory capacity and compute capability. In this work, we use model compression techniques to bridge this gap. We define the constraints imposed on the RNN by the HW and describe a method to satisfy them. Although model compression techniques are an active area of research, we are the…
Non-overlapping, Schwarz-type Domain Decomposition Method for Physics and Equality Constrained Artificial Neural Networks
organization= Department of Mechanical Engineering；Materials Science, University of Pittsburgh , city= Pittsburgh , postcode= 15261 , state= PA , country= USA
Artificial neural networks have successfully tackled a large variety of problems by training extremely deep networks via back-propagation. A direct application of back-propagation to spiking neural networks contains biologically implausible components, like the weight transport problem or separate inference and learning phases. Various methods address different components individually, but a complete solution remains intangible. Here, we take an alternate approach that avoids back-propagation and its associated issues entirely. Recent work in deep learning proposed independently training each …
