We consider approximation rates of sparsely connected deep rectified linear unit (ReLU) and rectified power unit (RePU) neural networks for functions in Besov spaces $B^α_{q}(L^p)$ in arbitrary dimension $d$, on general domains. We show that \alert{deep rectifier} networks with a fixed activation function attain optimal or near to optimal approximation rates for functions in the Besov space $B^α_τ(L^τ)$ on the critical embedding line $1/τ=α/d+1/p$ for \emph{arbitrary} smoothness order $α>0$. Using interpolation theory, this implies that the entire range of smoothness classes at or above the critical line is (near to) optimally approximated by deep ReLU/RePU networks.
Binary representation is desirable for its memory efficiency, computation speed and robustness. In this paper, we propose adjustable bounded rectifiers to learn binary representations for deep neural networks. While hard constraining representations across layers to be binary makes training unreasonably difficult, we softly encourage activations to diverge from real values to binary by approximating step functions. Our final representation is completely binary. We test our approach on MNIST, CIFAR10, and ILSVRC2012 dataset, and systematically study the training dynamics of the binarization pro…
Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations
Department of Mathematical Sciences, Tsinghua University（清华大学数学科学系）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；AI for Science Institute, Beijing（北京人工智能科学研究院）
Results on asymptotic characteristics of classes of functions with mixed smoothness are obtained in the paper. Our main interest is in estimating the Kolmogorov widths of classes with small mixed smoothness. We prove the corresponding bounds for the unit balls of the trigonometric polynomials with frequencies from a hyperbolic cross. We demonstrate how our results on the Kolmogorov widths imply new upper bounds for the optimal sampling recovery in the $L_2$ norm of functions with small mixed smoothness.
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
We explore the phase diagram of approximation rates for deep neural networks and prove several new theoretical results. In particular, we generalize the existing result on the existence of deep discontinuous phase in ReLU networks to functional classes of arbitrary positive smoothness, and identify the boundary between the feasible and infeasible rates. Moreover, we show that all networks with a piecewise polynomial activation function have the same phase diagram. Next, we demonstrate that standard fully-connected architectures with a fixed width independent of smoothness can adapt to smoothne…
We give the theorem of coincidence of a class of functions defined by a generalised modulus of smoothness with a class of functions defined by the order of the best approximation by algebraic polynomials. We also prove the appropriate inverse theorem in approximation theory.
In this paper, a $k$-th generalized modulus of smoothness is defined based on an asymmetric operator of generalized translation and a theorem is proved about the coincidence of class of functions defined by this modulus and a class of functions having given order of best approximation by algebraic polynomials.
In the present paper, we use a generalised shift operator in order to define a generalised modulus of smoothness. By its means, we define generalised Lipschitz classes of functions, and we give their constructive characteristics. Specifically, we prove certain direct and inverse types theorems in approximation theory for best approximation by algebraic polynomials.
For $1<p\leqslant \infty$, we study the complexity and the existence of universal spaces for two classes of separable Banach spaces, denoted $\textsf{A}_p$ and $\textsf{N}_p$, and related to asymptotic smoothness in Banach spaces. We show that each of these classes is Borel in the class of separable Banach spaces. Then we build small families of Banach spaces that are both injectively and surjectively universal for these classes. Finally, we prove the optimality of this universality result, by proving in particular that none of these classes admits a universal space.
Kolmogorov widths of an intersection of Besov classes with dominating mixed smoothness in a Besov space
Faculty of Mechanics and Mathematics, Lomonosov Moscow State University（莫斯科国立大学力学与数学系）；Moscow Center for Fundamental and Applied Mathematics（莫斯科基础与应用数学中心）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of $\varepsilon$ uniformly over the interval. We show that shallow networks (i.e., ne…
In this paper we present results on asymptotic characteristics of multivariate function classes in the uniform norm. Our main interest is the approximation of functions with mixed smoothness parameter not larger than $1/2$. Our focus will be on the behavior of the best $m$-term trigonometric approximation as well as the decay of Kolmogorov and entropy numbers in the uniform norm. It turns out that these quantities share a few fundamental abstract properties like their behavior under real interpolation, such that they can be treated simultaneously. We start with proving estimates on finite rank…
We give a new characterization of Nikolskii-Besov classes of functions of fractional smoothness by means of a nonlinear integration by parts formula in the form of a nonlinear inequality. A similar characterization is obtained for Nikolskii-Besov classes with respect to Gaussian measures on finite- and infinite-dimensional spaces.
In this paper, approximation by means of algebraic polynomials of classes of functions defined by a generalised modulus of smoothness of operators of differentiation of these functions is considered. We give structural characteristics of classes of functions defined by the order of best approximation by algebraic polynomials.
In this paper we obtain the necessary and sufficient conditions for embedding results of different function classes. The main result is a criterion for embedding theorems for the so-called generalized Weyl-Nikol'skii class and the generalized Lipschitz class. To define the Weyl-Nikol'skii class, we use the concept of a $(λ,β)$-derivative, which is a generalization of the derivative in the sense of Weyl. As corollaries, we give estimates of norms and moduli of smoothness of transformed Fourier series.
In this article, we introduce classes of functions whose increment is controlled by the measure of a ball containing the corresponding points and a nonnegative function p(.) that is summable with respect to measure. These classes of functions can be considered as spaces with variable smoothness depending on the structure of the measure in a neighborhood of a given point. Moreover, we present several descriptions generalized classes of variable exponent Hajłasz-Sobolev type on metric measure spaces by various maximal functions and we establish the equivalence between them.
This article is devoted to nonlinear approximation and estimation via piecewise polynomials built on partitions into dyadic rectangles. The approximation rate is studied over possibly inhomogeneous and anisotropic smoothness classes that contain Besov classes. Highlighting the interest of such a result in statistics, adaptation in the minimax sense to both inhomogeneity and anisotropy of a related multivariate density estimator is proved. Besides, that estimation procedure can be implemented with a computational complexity simply linear in the sample size.
We study fractional smoothness of measures on $\mathbb{R}^k$, that are images of a Gaussian measure under mappings from Gaussian Sobolev classes. As a consequence we obtain Nikolskii--Besov fractional regularity of these distributions under some weak nondegeneracy assumption.
