Neural networks are regularly employed in adaptive control of nonlinear systems and related methods of reinforcement learning. A common architecture uses a neural network with a single hidden layer (i.e. a shallow network), in which the weights and biases are fixed in advance and only the output layer is trained. While classical results show that there exist neural networks of this type that can approximate arbitrary continuous functions over bounded regions, they are non-constructive, and the networks used in practice have no approximation guarantees. Thus, the approximation properties required for control with neural networks are assumed, rather than proved. In this paper, we aim to fill this gap by showing that for sufficiently smooth functions, ReLU networks with randomly generated weights and biases achieve $L_{\infty}$ error of $O(m^{-1/2})$ with high probability, where $m$ is the number of neurons. It suffices to generate the weights uniformly over a sphere and the biases uniformly over an interval. We show how the result can be used to get approximations of required accuracy in a model reference adaptive control application.
Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
Institute of Natural Sciences & School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学自然科学研究院与数学科学学院);Computer Science and Engineering, Ohio State University(俄亥俄州立大学计算机科学与工程系);Electrical and Systems Engineering & Department of Radiology, University of Pennsylvania(宾夕法尼亚大学电气与系统工程系及放射学系)
Normalized gradient flow optimization in the training of ReLU artificial neural networks
Basque Center for Applied Mathematics(巴斯克应用数学中心);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Münster(明斯特大学);University of Duisburg-Essen(杜伊斯堡-埃森大学)
On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector
School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)(数据科学学院和人工智能学院,香港中文大学(深圳));Applied Mathematics: Institute for Analysis and Numerics, Faculty of Mathematics and Computer Science, University of Münster(应用数学:分析与数值研究所,数学与计算机科学学院,穆恩斯特大学)
On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);University of Münster(明斯特大学)
在这篇文章中,我们研究了具有任意大量隐藏层的全连接前馈深度ReLU ANN,并且我们证明了在所考虑的监督学习问题的输入数据的概率分布的未归一化密度函数是分段多项式的假设下,在这种ANN的训练中具有随机初始化的GD最优化方法的风险的收敛性,假设目标函数(描述输入数据和输出数据之间的关系)是分段多项式,并且假设所考虑的监督学习问题的风险函数至少允许一个正则全局最小值。此外,在只有一个隐藏层和一维输入的浅人工神经网络的特殊情况下,我们也验证了这一假设,证明在训练这样的浅人工神经网络,对于每个Lipschitz连续目标函数存在一个全局最小值的风险景观。最后,在使用ReLU激活的深度ANN的训练中,我们还研究了梯度流(GF)微分方程的解,并证明了每个非发散GF轨迹都以多项式收敛速率收敛到临界点(在限制Fréchet次可微性的意义上)。我们的数学收敛分析建立在我们以前的文章Eberle等人的思想基础上,基于来自实代数几何的工具,例如半代数函数和广义Kurdyka-Lojasiewicz不等式的概念,基于来自泛函分析的工具,例如Arzelà-Ascoli定理,基于来自非光滑分析的工具,例如极限Fréchet次梯度的概念,以及具有固定架构的浅ReLU ANN的实现函数的集合形成连续函数的集合的闭子集的事实Petersen et al.
Optimization with Region-Reduced ReLU Neural Networks
Max Planck Institute for Dynamics of Complex Technical Systems(马克斯·普朗克复杂技术系统动力学研究所);Otto von Guericke University(奥托·冯·格里克大学);Leibniz University Hannover(汉诺威莱布尼茨大学)