首页/论文/2001.03040

arXiv 2001.030402020-01cs.LGcs.NAmath.NAstat.ML

Deep Network Approximation for Smooth Functions

National University of Singapore（新加坡国立大学）；Purdue University（普渡大学）

Jianfeng Lu, Zuowei Shen, Haizhao Yang, Shijun Zhang

发表信息：: SIAM Journal on Mathematical Analysis, Volume 53, Issue 5, September 2021, Pages 5465-5506
DOI：: 10.1137/20M134695X

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N)$ and depth $\mathcal{O}(L)$ with an approximation error $\mathcal{O}(N^{-L})$. Through local Taylor expansions and their deep ReLU network approximations, we show that deep ReLU networks of width $\mathcal{O}(N\ln N)$ and depth $\mathcal{O}(L\ln L)$ can approximate $f\in C^s([0,1]^d)$ with a nearly optimal approximation error $\mathcal{O}(\|f\|_{C^s([0,1]^d)}N^{-2s/d}L^{-2s/d})$. Our estimate is non-asymptotic in the sense that it is valid for arbitrary width and depth specified by $N\in\mathbb{N}^+$ and $L\in\mathbb{N}^+$, respectively.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Approximation of Smoothness Classes by Deep Rectifier Networks

Centrale Nantes（南特中央理工学院）；LMJL（南特让·勒雷数学实验室）

We consider approximation rates of sparsely connected deep rectified linear unit (ReLU) and rectified power unit (RePU) neural networks for functions in Besov spaces $B^α_{q}(L^p)$ in arbitrary dimension $d$, on general domains. We show that \alert{deep rectifier} networks with a fixed activation function attain optimal or near to optimal approximation rates for functions in the Besov space $B^α_τ(L^τ)$ on the critical embedding line $1/τ=α/d+1/p$ for \emph{arbitrary} smoothness order $α>0$. Using interpolation theory, this implies that the entire range of smoothness classes at or above the cr…

2021-10-29cs.LGcs.NAmath.FA

Smooth function approximation by deep neural networks with general activation functions

Seoul National University（首尔大学）

There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functions. This class of activation functions includes most of frequently used activation functions. We derive the required depth, width and sparsity of a deep neural network to approximate any Hölder smooth function upto a given approximation error for the large class of activation fu…

2019-06-27cs.LGstat.ML

Value Function Approximation in Noisy Environments Using Locally Smoothed Regularized Approximate Linear Programs

United States Naval Academy（美国海军学院）；Duke University（杜克大学）

Recently, Petrik et al. demonstrated that L1Regularized Approximate Linear Programming (RALP) could produce value functions and policies which compared favorably to established linear value function approximation techniques like LSPI. RALP's success primarily stems from the ability to solve the feature selection and value function approximation steps simultaneously. RALP's performance guarantees become looser if sampled next states are used. For very noisy domains, RALP requires an accurate model rather than samples, which can be unrealistic in some practical scenarios. In this paper, we demon…

2012-10-16cs.LGstat.ML

Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space

The University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Kyushu Institute of Technology（九州工业大学）

Deep learning has exhibited superior performance for various tasks, especially for high-dimensional datasets, such as images. To understand this property, we investigate the approximation and estimation ability of deep learning on anisotropic Besov spaces. The anisotropic Besov space is characterized by direction-dependent smoothness and includes several function classes that have been investigated thus far. We demonstrate that the approximation error and estimation error of deep learning only depend on the average value of the smoothness parameters in all directions. Consequently, the curse o…

2021-09-30cs.LGstat.ML

连接光滑性与逼近：图神经网络中过平滑的理论洞见

Bridging Smoothness and Approximation: Theoretical Insights into Over-Smoothing in Graph Neural Networks

South China Agricultural University（华南农业大学）；City University of Hong Kong（香港城市大学）；Zhejiang Normal University（浙江师范大学）；Zhejiang Institute of Optoelectronics（浙江光电研究所）；The University of Sydney（悉尼大学）

本文基于图上 $K$-泛函与光滑性模的等价性，建立 GCN 逼近下界并解释高频能量衰减导致的过平滑，实验验证了指数衰减阶理论。

2024-08-05cs.AIcs.LGmath.FA

Non-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximation

University at Buffalo（布法罗大学）；University of Utah（犹他大学）

Temporal-difference learning with gradient correction (TDC) is a two time-scale algorithm for policy evaluation in reinforcement learning. This algorithm was initially proposed with linear function approximation, and was later extended to the one with general smooth function approximation. The asymptotic convergence for the on-policy setting with general smooth function approximation was established in [bhatnagar2009convergent], however, the finite-sample analysis remains unsolved due to challenges in the non-linear and two-time-scale update structure, non-convex objective function and the tim…

2021-10-28cs.LG

更多相关论文收起更多论文

Approximation smooth and sparse functions by deep neural networks without saturation

Xi'an University of Technology（西安理工大学）

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks (deep nets for short) with three hidden layers to approximate smooth and sparse functions. In particular, we prove that the constructed deep nets can reach the optimal approximation rate in approximating both smooth and sparse functions with controllable magnitude of free parameters. Since the saturation that describes the bottleneck of approximate is an insurmountable problem of constructive neural networks, we also p…

2020-01-13cs.ITcs.LGmath.IT

最优近似——软最大函数的平滑性权衡

Optimal Approximation -- Smoothness Tradeoffs for Soft-Max Functions

Google Research（谷歌研究）；Massachusetts Institute of Technology（麻省理工学院）

本文提出分段线性和功率机制，分别在最坏情况加性近似与ℓq-范数平滑度、期望乘法近似与Rényi散度平滑度之间实现最优权衡，适用于机器学习和机制设计领域。

2025-12-30cs.DScs.LG

Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint

Hong Kong Baptist University（香港浸会大学）；Georgia Tech（佐治亚理工学院）；The Hong Kong University of Science and Technology（香港科技大学）；Google Research（谷歌研究院）

Overparameterized neural networks enjoy great representation power on complex data, and more importantly yield sufficiently smooth output, which is crucial to their generalization and robustness. Most existing function approximation theories suggest that with sufficiently many parameters, neural networks can well approximate certain classes of functions in terms of the function value. The neural network themselves, however, can be highly nonsmooth. To bridge this gap, we take convolutional residual networks (ConvResNets) as an example, and prove that large ConvResNets can not only approximate …

2022-06-09cs.LGstat.ML

非凸光滑性条件的最新进展及其在深度线性神经网络中的适用性

Recent Advances in Non-convex Smoothness Conditions and Applicability to Deep Linear Neural Networks

University of Wisconsin（威斯康星大学）

本文综述了深度学习非凸优化中的新光滑性条件，对其进行排序并给出判定准则，评估其在深度线性神经网络二分类训练中的适用性。

2024-09-20cs.LGmath.OC

舍入函数的光滑近似

Smooth Approximations of the Rounding Function

本文提出了经典舍入函数的两种新型光滑近似方法，适用于可微优化与机器学习，通过局部化sigmoid窗口和sigmoid导数加权和实现，在锐度参数趋于无穷时逐点收敛，且计算成本低、精度高，为硬舍入提供了完全可微的替代方案。

2025-04-26cs.LGmath.OC

Addressing Over-Smoothing in Graph Neural Networks via Deep Supervision

CSIRO’s DATA61（澳大利亚联邦科学与工业研究组织DATA61部门）

Learning useful node and graph representations with graph neural networks (GNNs) is a challenging task. It is known that deep GNNs suffer from over-smoothing where, as the number of layers increases, node representations become nearly indistinguishable and model performance on the downstream task degrades significantly. To address this problem, we propose deeply-supervised GNNs (DSGNNs), i.e., GNNs enhanced with deep supervision where representations learned at all layers are used for training. We show empirically that DSGNNs are resilient to over-smoothing and can outperform competitive bench…

2022-02-25cs.LG

SAU: Smooth activation function using convolution with approximate identities

Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.12% improvement with Shuf…

2021-09-27cs.AIcs.CVcs.LG

SMU: smooth activation function for deep networks using smoothing maximum technique

Deep learning researchers have a keen interest in proposing two new novel activation functions which can boost network performance. A good choice of activation function can have significant consequences in improving network performance. A handcrafted activation is the most common choice in neural network models. ReLU is the most common choice in the deep learning community due to its simplicity though ReLU has some serious drawbacks. In this paper, we have proposed a new novel activation function based on approximation of known activation functions like Leaky ReLU, and we call this function Sm…

2022-04-11cs.AIcs.CVcs.LG