Deep learning, a multi-layered neural network approach inspired by the brain, has revolutionized machine learning. One of its key enablers has been backpropagation, an algorithm that computes the gradient of a loss function with respect to the weights and biases in the neural network model, in combination with its use in gradient descent. However, the implementation of deep learning in digital computers is intrinsically energy hungry, with energy consumption becoming prohibitively high for many applications. This has stimulated the development of specialized hardware, ranging from neuromorphic CMOS integrated circuits and integrated photonic tensor cores to unconventional, material-based computing system. The learning process in these material systems, realized, e.g., by artificial evolution, equilibrium propagation or surrogate modelling, is a complicated and time-consuming process. Here, we demonstrate a simple yet efficient and accurate gradient extraction method, based on the principle of homodyne detection, for performing gradient descent on a loss function directly in a physical system without the need of an analytical description. By perturbing the parameters that need to be optimized using sinusoidal waveforms with distinct frequencies, we effectively obtain the gradient information in a highly robust and scalable manner. We illustrate the method in dopant network processing units, but argue that it is applicable in a wide range of physical systems. Homodyne gradient extraction can in principle be fully implemented in materia, facilitating the development of autonomously learning material systems.
eGAD! double descent is explained by Generalized Aliasing Decomposition
Department of Physics and Astronomy, Brigham Young University, Provo, Utah 84602, USA(物理与天文学系, BYU,犹他州普罗沃,84602,USA);Department of Mathematics, Brigham Young University, Provo, Utah 84602, USA(数学系, BYU,犹他州普罗沃,84602,USA)
Neural Network Surrogate and Projected Gradient Descent for Fast and Reliable Finite Element Model Calibration: a Case Study on an Intervertebral Disc
Institute for Neuroradiology, TUM University Hospital, School of Medicine and Health, Technical University of Munich (TUM)(慕尼黑工业大学神经放射学研究所,慕尼黑工业大学医院,医学与健康学院);Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital(慕尼黑工业大学医疗保健与医学人工智能主席席位,慕尼黑工业大学医院);Department of Trauma and Reconstructive Surgery, University Hospital Halle, Martin-Luther-University Halle-Wittenberg(马丁路德大学哈雷-维滕贝格分校哈雷大学医院创伤与重建外科);Department of Mechanical Engineering, Federal University of Santa Catarina(圣卡塔琳娜联邦大学机械工程系);Department of Mechanical Engineering, Federal University of Santa Maria(圣玛丽亚联邦大学机械工程系);Associate Professorship of Sport Equipment and Sport Materials, School of Engineering and Design, Technical University of Munich(慕尼黑工业大学工程与设计学院运动装备与运动材料副教授席位);Department of Computing, Imperial College London(伦敦帝国理工学院计算机系)
Mohamed bin Zayed University of Artificial Intelligence(莫萨德·本·泽德人工智能大学);University at Buffalo(布法罗大学);King Abdullah University of Science and Technology(卡布斯国王科学与技术大学)