Intelligent gradient amplification for deep neural networks
Georgia State University（佐治亚州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；Chinese Academy of Sciences（中国科学院）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）
Sunitha Basodi, Krishna Pusuluri, Xueli Xiao, Yi Pan
Deep learning models offer superior performance compared to other machine learning techniques for a variety of tasks and domains, but pose their own challenges. In particular, deep learning models require larger training times as the depth of a model increases, and suffer from vanishing gradients. Several solutions address these problems independently, but there have been minimal efforts to identify an integrated solution that improves the performance of a model by addressing vanishing gradients, as well as accelerates the training process to achieve higher performance at larger learning rates. In this work, we intelligently determine which layers of a deep learning model to apply gradient amplification to, using a formulated approach that analyzes gradient fluctuations of layers during training. Detailed experiments are performed for simpler and deeper neural networks using two different intelligent measures and two different thresholds that determine the amplification layers, and a training strategy where gradients are amplified only during certain epochs. Results show that our amplification offers better performance compared to the original models, and achieves accuracy improvement of around 2.5% on CIFAR- 10 and around 4.5% on CIFAR-100 datasets, even when the models are trained with higher learning rates.
When majority rules, minority loses: bias amplification of gradient descent
University of Lille（里尔大学）；Toulouse School of Economics（图卢兹经济学院）；Institut Universitaire de France (IUF)（法国大学研究院（IUF））；ANITI；Université de Toulouse（图卢兹大学）；INRIA（法国国家信息与自动化研究所）
Intersection of Reinforcement Learning and Bayesian Optimization for Intelligent Control of Industrial Processes: A Safe MPC-based DPG using Multi-Objective BO
Department of Mechanical Engineering, Clemson University（机械工程系，克莱姆森大学）
Reconfigurable Intelligent Surface Aided Vehicular Edge Computing: Joint Phase-shift Optimization and Multi-User Power Allocation
School of Internet of Things Engineering, Jiangnan University（江南大学物联网工程学院）；Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学电子工程系，北京信息科学与技术国家研究中心）；State Key Lab. of ISN and School of Telecommunications Engineering, Xidian University（西安电子科技大学信息与通信工程学院）；Department of Electrical and Computer Engineering, Hong Kong University of Science and Technology (HKUST)（香港科学与技术大学电气与计算机工程系）
