In recent years, functional neural networks have been proposed and studied in order to approximate nonlinear continuous functionals defined on $L^p([-1, 1]^s)$ for integers $s\ge1$ and $1\le p<\infty$. However, their theoretical properties are largely unknown beyond universality of approximation or the existing analysis does not apply to the rectified linear unit (ReLU) activation function. To fill in this void, we investigate here the approximation power of functional deep neural networks associated with the ReLU activation function by constructing a continuous piecewise linear interpolation under a simple triangulation. In addition, we establish rates of approximation of the proposed functional deep ReLU networks under mild regularity conditions. Finally, our study may also shed some light on the understanding of functional data learning algorithms.
A Momentum Accelerated Algorithm for ReLU-based Nonlinear Matrix Decomposition
School of Mathematics and Computational Science, Xiangtan University（湘潭大学数学与计算科学学院）；School of Mathematical Sciences, Beihang University（北京航空航天大学数学科学学院）
Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks
Elmore Family School of Electrical and Computer Engineering（埃尔莫尔电气与计算机工程学院）；Purdue University（普渡大学）；Department of Physics and Astronomy（物理与天文学系）；Halıcıoğlu Data Science Institute（哈利奇奥格卢数据科学研究所）；University of California San Diego（加州大学圣地亚哥分校）；Rosen Center For Advanced Computing（罗森先进计算中心）
Deep neural networks with ReLU, leaky ReLU, and softplus activation provably overcome the curse of dimensionality for Kolmogorov partial differential equations with Lipschitz nonlinearities in the $L^p$-sense
Department of Mathematics & Informatics, University of Wuppertal（乌尔姆大学数学与信息学系）；School of Data Science and Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳）数据科学学院及深圳大数据研究院）；Applied Mathematics: Institute for Analysis and Numerics, University of Münster（穆恩斯特大学应用数学系及分析与数值分析研究所）；Department of Mathematical Sciences, University of Arkansas（阿肯色大学数学科学系）；Center for Astrophysics, Space Physics, and Engineering Research, Baylor University（贝勒大学天体物理学、空间物理学与工程研究中心）
Department of Computer Science Rensselaer Polytechnic Institute（计算机科学系伦塞拉尔理工大学）；Department of Biomedical Engineering Rensselaer Polytechnic Institute（生物医学工程系伦塞拉尔理工大学）
Reachable Set Computation and Safety Verification for Neural Networks with ReLU Activations
Vanderbilt University（范德堡大学）
神经网络已被广泛用于解决复杂的现实问题。由于神经网络的复杂性、非线性和非凸性，对神经网络输出行为的形式安全保证将是其在安全关键系统中应用的关键，本文研究了一类由ReLU（Rectified Linear Unit）激活函数构成的神经网络的输出可达集计算和安全性验证问题.提出了一种逐层计算输出可达集的方法。计算公式化的形式，一组操作的多面体，它可以有效地应用与援助的多面体计算工具。基于输出可达集计算结果，可以通过检查由多面体的并集描述的不安全区域和输出可达集的交集来执行ReLU神经网络的安全性验证。一个随机生成的ReLU神经网络的数值例子表明本文开发的方法的有效性。
Deep neural networks with ReLU, leaky ReLU, and softplus activation provably overcome the curse of dimensionality for space-time solutions of semilinear partial differential equations
University of Wuppertal（伍珀塔尔大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Münster（明斯特大学）；Toyota Financial Services（丰田金融服务公司）
Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
Institute of Natural Sciences & School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学自然科学研究院与数学科学学院）；Computer Science and Engineering, Ohio State University（俄亥俄州立大学计算机科学与工程系）；Electrical and Systems Engineering & Department of Radiology, University of Pennsylvania（宾夕法尼亚大学电气与系统工程系及放射学系）
An Accelerated Alternating Partial Bregman Algorithm for ReLU-based Matrix Decomposition
School of Mathematics and Computational Science, Xiangtan University（湘潭大学数学与计算科学学院）；School of Mathematical Sciences, Beihang University（北京航空航天大学数学科学学院）
Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference
Chongqing Key Laboratory of Secure Computing for Biology（重庆生物安全计算重点实验室）；Chongqing Institute of Green and Intelligent Technology（重庆绿色智能技术研究所）；Chinese Academy of Sciences（中国科学院）；Department of Industrial and Systems Engineering（工业与系统工程系）
Normalized gradient flow optimization in the training of ReLU artificial neural networks
Basque Center for Applied Mathematics（巴斯克应用数学中心）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Münster（明斯特大学）；University of Duisburg-Essen（杜伊斯堡-埃森大学）
Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity
Department of Computer Science University of Wisconsin-Madison（计算机科学系威斯康星大学麦迪逊分校）；Department of Electrical & Computer Engineering University of Wisconsin-Madison（电气与计算机工程系威斯康星大学麦迪逊分校）
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China（数学科学国家重点实验室，数学与系统科学研究院，中国科学院，北京100190，中国）；School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China（中国科学院大学数学科学学院，北京100049，中国）；Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China（上海数学与交叉科学研究所（SIMIS），上海，中国）；Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China（浙江省系统健康科学重点实验室，生命科学学院，杭州先进研究院，中国科学院大学，中国科学院，杭州310024，中国）
Implicit Regularization in ReLU Networks with the Square Loss
Weizmann Institute of Science（魏茨曼科学研究所）
理解梯度下降的隐式正则化（或隐式偏差）是最近一个非常活跃的研究领域。然而，非线性神经网络中的隐式正则化仍然知之甚少，特别是对于回归损失，如平方损失。也许令人惊讶的是，我们证明了即使对于单个ReLU神经元，也不可能通过模型参数的任何显式函数来表征具有平方损失的隐式正则化（尽管从积极的方面来看，我们可以近似地表征它）。对于一个隐藏层网络，我们证明了一个类似的结果，一般来说，不可能以这种方式表征隐式正则化属性，除了Du et al. [2018]中确定的“平衡性”属性。我们的研究结果表明，可能需要一个更一般的框架比迄今为止考虑的理解隐式正则化的非线性预测，并提供了一些线索，这个框架应该是什么。
