We present results of numerical experiments for neural networks with stochastic gradient-based optimization with adaptive momentum. This widely applied optimization has proved convergence and practical efficiency, but for long-run training becomes numerically unstable. We show that numerical artifacts are observable not only for large-scale models and finally lead to divergence also for case of shallow narrow networks. We argue this theory by experiments with more than 1600 neural networks trained for 50000 epochs. Local observations show presence of the same behavior of network parameters in both stable and unstable training segments. Geometrical behavior of parameters forms double twisted spirals in the parameter space and is caused by alternating of numerical perturbations with next relaxation oscillations in values for 1st and 2nd momentum.
Adaptive digital twins for predictive decision-making: Online Bayesian learning of transition dynamics
MOX – Dipartimento di Matematica, Politecnico di Milano(MOX——数学系,米兰理工大学);Dipartimento di Ingegneria Civile e Ambientale, Politecnico di Milano(土木工程与环境工程系,米兰理工大学);Mathematics Department, Emory University(埃默里大学数学系)
Adaptive Federated Learning and Digital Twin for Industrial Internet of Things
Northwestern Polytechnical University(西北工业大学);Xidian University(西安电子科技大学);University of Oslo(奥斯陆大学);Simula Metropolitan Center for Digital Engineering(西穆拉都会数字工程中心)
Adaptive Sensor Steering Strategy Using Deep Reinforcement Learning for Dynamic Data Acquisition in Digital Twins
Dynamic Research Group(动态研究组);School of Mechanical, Aerospace and Civil Engineering(机械、航空航天与土木工程学院);University of Sheffield(谢菲尔德大学);Siemens Digital Industries Software NV(西门子数字工业软件公司);The Alan Turing Institute(艾伦·图灵研究所)
ANSR-DT: A Neuro-Symbolic Framework for Adaptive and Explainable Digital Twins
Department of Information Systems, University of Maryland Baltimore County(信息系统系,马里兰大学巴尔的摩县分校);Department of Computer Science and Engineering, University at Buffalo(计算机科学与工程系,布法罗大学);Department of Computer Science, University of Colorado Boulder(计算机科学系,科罗拉多大学波德分校)
PDx -- Adaptive Credit Risk Forecasting Model in Digital Lending using Machine Learning Operations
OM & QT Area, Indian Institute of Management Indore(印度管理学院印度尔分校OM与QT部门);Department of Statistics & Data Science, National University of Singapore(新加坡国立大学统计与数据科学系)
Can machine learning solve the challenge of adaptive learning and the individualization of learning paths? A field experiment in an online learning platform