Understanding the power of depth in feed-forward neural networks is an ongoing challenge in the field of deep learning theory. While current works account for the importance of depth for the expressive power of neural-networks, it remains an open question whether these benefits are exploited during a gradient-based optimization process. In this work we explore the relation between expressivity properties of deep networks and the ability to train them efficiently using gradient-based algorithms. We give a depth separation argument for distributions with fractal structure, showing that they can be expressed efficiently by deep networks, but not with shallow ones. These distributions have a natural coarse-to-fine structure, and we show that the balance between the coarse and fine details has a crucial effect on whether the optimization process is likely to succeed. We prove that when the distribution is concentrated on the fine details, gradient-based algorithms are likely to fail. Using this result we prove that, at least in some distributions, the success of learning deep networks depends on whether the distribution can be well approximated by shallower networks, and we conjecture that this property holds in general.
Carnegie Mellon University（卡内基梅隆大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Maryland（马里兰大学）；Cornell Tech（康奈尔科技学院）
Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. However, the use of stochastic gradient descent combined with the nonconvexity of the underlying optimization problems renders parameter learning susceptible to initialization. To address this issue, a variety of methods that rely on random parameter initialization or knowledge distillation have been proposed in the past. In this paper, we propose FuseInit, a novel method to initialize shallower networks by fusing neighboring layers of deeper networks that are trained with random initia…
Laboratoire de Probabilités, Statistique et Modélisation（概率、统计与建模实验室）；Sorbonne Université（索邦大学）
Training a very deep neural network is a challenging task, as the deeper a neural network is, the more non-linear it is. We compare the performances of various preconditioned Langevin algorithms with their non-Langevin counterparts for the training of neural networks of increasing depth. For shallow neural networks, Langevin algorithms do not lead to any improvement, however the deeper the network is and the greater are the gains provided by Langevin algorithms. Adding noise to the gradient descent allows to escape from local traps, which are more frequent for very deep neural networks. Follow…
Derandomized shallow shadows: Efficient Pauli learning with bounded-depth circuits
Harvard University（哈佛大学）；Harvard-Smithsonian Center for Astrophysics（哈佛-史密森尼天体物理中心）；University of Maryland（马里兰大学）；Joint Quantum Institute（联合量子研究所）；Joint Center for Quantum Information and Computer Science（量子信息与计算机科学联合中心）；National Institute of Standards and Technology（美国国家标准与技术研究院）；University of Innsbruck（因斯布鲁克大学）
Institute of Mathematical Sciences, Claremont Graduate University（克莱蒙特研究生大学数学科学学院）；Center for Brains, Minds, and Machines, McGovern Institute for Brain Research, Massachusetts Institute of Technology（麻省理工学院麦戈文脑研究所脑、心智与机器中心）
This paper is motivated by an open problem around deep networks, namely, the apparent absence of over-fitting despite large over-parametrization which allows perfect fitting of the training data. In this paper, we analyze this phenomenon in the case of regression problems when each unit evaluates a periodic activation function. We argue that the minimal expected value of the square loss is inappropriate to measure the generalization error in approximation of compositional functions in order to take full advantage of the compositional structure. Instead, we measure the generalization error in t…
Deep neural networks are over-parameterized, which implies that the number of parameters are much larger than the number of samples used to train the network. Even in such a regime deep architectures do not overfit. This phenomenon is an active area of research and many theories have been proposed trying to understand this peculiar observation. These include the Vapnik Chervonenkis (VC) dimension bounds and Rademacher complexity bounds which show that the capacity of the network is characterized by the norm of weights rather than the number of parameters. However, the effect of input noise on …
The University of Western Australia（西澳大利亚大学）；The University of New South Wales（新南威尔士大学）
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a single hidden layer. The transformations of both deep rectifier nets and deep residual nets are conducted to demonstrate the advantages of the residual nets over the conventional neural nets and the advantages of the deep neural nets over the shallow neural nets. In summary, for …
It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones. We shed light on this by proving that the total number of neurons $m$ required to approximate natural classes of multivariate polynomials of $n$ variables grows only linearly with $n$ for deep neural networks, but grows exponentially when merely a single hidden layer is allowed. We also provide evidence that when the number of hidden layers is increased from $1$ to $k$, the neuron requirement grows exponentially not with $n$ but with $n^{1/k}$, s…
The Hong Kong Polytechnic University（香港理工大学）；ETH Zurich（苏黎世联邦理工学院）
Many interesting problems in machine learning are being revisited with new deep learning tools. For graph-based semisupervised learning, a recent important development is graph convolutional networks (GCNs), which nicely integrate local vertex features and graph topology in the convolutional layers. Although the GCN model compares favorably with other state-of-the-art methods, its mechanisms are not clear and it still requires a considerable amount of labeled data for validation and model selection. In this paper, we develop deeper insights into the GCN model and address its fundamental limits…
Indian Institute of Information Technology Sri City（斯里城印度信息技术学院）
The Convolutional Neural Networks (CNNs), in domains like computer vision, mostly reduced the need for handcrafted features due to its ability to learn the problem-specific features from the raw input data. However, the selection of dataset-specific CNN architecture, which mostly performed by either experience or expertise is a time-consuming and error-prone process. To automate the process of learning a CNN architecture, this paper attempts at finding the relationship between Fully Connected (FC) layers with some of the characteristics of the datasets. The CNN architectures, and recently data…
University of Toronto（多伦多大学）；Microsoft Research（微软研究院）
Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this extended abstract, we show that shallow feed-forward networks can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using a total number of parameters similar to the original deep model. We evaluate our method on the TIMIT phoneme recognition task and are able to train shallow fully-connected nets that perform simila…
We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger. This includes indicators of balls and ellipses; non-linear functions which are radial with respect to the $L_1$ norm; and smooth non-linear functions. We also show that these gaps can be observed experimentally: Increasing the depth indeed allows better learning than increasing width, when training neural networks to learn an indicat…
An additive autoencoder for dimension reduction, which is composed of a serially performed bias estimation, linear trend estimation, and nonlinear residual estimation, is proposed and analyzed. Computational experiments confirm that an autoencoder of this form, with only a shallow network to encapsulate the nonlinear behavior, is able to identify an intrinsic dimension of a dataset with a low autoencoding error. This observation leads to an investigation in which shallow and deep network structures, and how they are trained, are compared. We conclude that the deeper network structures obtain l…
TU München（慕尼黑工业大学）；Czech Technical University in Prague（捷克理工大学）；Duke University（杜克大学）
We address the structure identification and the uniform approximation of sums of ridge functions $f(x)=\sum_{i=1}^m g_i(a_i\cdot x)$ on ${\mathbb R}^d$, representing a general form of a shallow feed-forward neural network, from a small number of query samples. Higher order differentiation, as used in our constructive approximations, of sums of ridge functions or of their compositions, as in deeper neural network, yields a natural connection between neural network weight identification and tensor product decomposition identification. In the case of the shallowest feed-forward neural network, se…
Asymptotic convexity of wide and shallow neural networks
Department of Electrical Engineering (retd.) Indian Institute of Technology Bombay Mumbai 400076, India. Email（电气工程系（退休）印度理工学院班加罗尔 孟买 400076, 印度。电子邮件）；Centre for Machine Intelligence（人工智能中心）；Data Science Indiaan Institute of Technology Bombay Mumbai 400076, India. Email（数据科学 印度理工学院班加罗尔 孟买 400076, 印度。电子邮件）
RIKEN Center for Emergent Matter Science (CEMS)（理化学研究所涌现物质科学中心（CEMS））；The University of Tokyo（东京大学）
It has been recognized that a heavily overparameterized artificial neural network exhibits surprisingly good generalization performance in various machine-learning tasks. Recent theoretical studies have made attempts to unveil the mystery of the overparameterization. In most of those previous works, the overparameterization is achieved by increasing the width of the network, while the effect of increasing the depth has remained less well understood. In this work, we investigate the effect of increasing the depth within an overparameterized regime. To gain an insight into the advantage of depth…
Towards a deeper GCN: Alleviate over-smoothing with iterative training and fine-tuning
Institute of Big Data Science and Industry（大数据科学与产业研究院）；Shanxi University（山西大学）；Key Laboratory of Evolutionary Science Intelligence of Shanxi Province（山西省进化科学智能重点实验室）；Zhengzhou University of Aeronautics（郑州航空工业大学）
Towards the Training of Deeper Predictive Coding Neural Networks
Institute of Logic and Computation, Vienna University of Technology（逻辑与计算研究所，维也纳技术大学）；University of Oxford（牛津大学）；VERSES AI Research Lab（VERSES AI研究实验室）
We establish, for the first time, connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks corresp…
Universitat Politecnica de Catalunya（加泰罗尼亚理工大学）；Insight Center for Data Analytics（洞察数据分析中心）；Dublin City University（都柏林城市大学）
The prediction of salient areas in images has been traditionally addressed with hand-crafted features based on neuroscience principles. This paper, however, addresses the problem with a completely data-driven approach by training a convolutional neural network (convnet). The learning process is formulated as a minimization of a loss function that measures the Euclidean distance of the predicted saliency map with the provided ground truth. The recent publication of large datasets of saliency prediction has provided enough data to train end-to-end architectures that are both fast and accurate. T…
KingSoft AI Lab（金山AI实验室）；University of Groningen（格罗宁根大学）
Learning deeper models is usually a simple and effective approach to improve model performance, but deeper models have larger model parameters and are more difficult to train. To get a deeper model, simply stacking more layers of the model seems to work well, but previous works have claimed that it cannot benefit the model. We propose to train a deeper model with recurrent mechanism, which loops the encoder and decoder blocks of Transformer in the depth direction. To address the increasing of model parameters, we choose to share parameters in different recursive moments. We conduct our experim…
Unlocking the Potential of Federated Learning for Deeper Models
Beihang University（北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Zhengzhou University（郑州大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；Lingnan University（岭南大学）
Naval Research Laboratory（海军研究实验室）；Naval Center for Applied Research in Artificial Intelligence（海军人工智能应用研究中心）；NRC Postdoctoral Fellow（NRC博士后研究员）
In person attributes recognition, we describe a person in terms of their appearance. Typically, this includes a wide range of traits including age, gender, clothing, and footwear. Although this could be used in a wide variety of scenarios, it generally is applied to video surveillance, where attribute recognition is impacted by low resolution, and other issues such as variable pose, occlusion and shadow. Recent approaches have used deep convolutional neural networks (CNNs) to improve the accuracy in person attribute recognition. However, many of these networks are relatively shallow and it is …
Confidence-gated training for efficient early-exit neural networks
International University of Rabat（拉巴特国际大学）；TICLab（TIC实验室）；Thales Alenia Space（泰勒斯阿莱尼亚空间）；Mohammed 6 Polytechnic University（穆莱·6技术大学）；University of Leeds（利兹大学）
We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.
Fine-tuning a network which has been trained on a large dataset is an alternative to full training in order to overcome the problem of scarce and expensive data in medical applications. While the shallow layers of the network are usually kept unchanged, deeper layers are modified according to the new dataset. This approach may not work for ultrasound images due to their drastically different appearance. In this study, we investigated the effect of fine-tuning different layers of a U-Net which was trained on segmentation of natural images in breast ultrasound image segmentation. Tuning the cont…
Borealis AI；University of Waterloo（滑铁卢大学）；University of Alberta（阿尔伯塔大学）；McGill University（麦吉尔大学）；Mila（米拉人工智能研究所）
It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train $48$ layers of transformers, comprising $24$ fine-tune…
University of Alberta（阿尔伯塔大学）；National Research Council of Canada（加拿大国家研究委员会）；American University（美利坚大学）；Virginia Commonwealth University（弗吉尼亚联邦大学）
Structural concept complexity, class overlap, and data scarcity are some of the most important factors influencing the performance of classifiers under class imbalance conditions. When these effects were uncovered in the early 2000s, understandably, the classifiers on which they were demonstrated belonged to the classical rather than Deep Learning categories of approaches. As Deep Learning is gaining ground over classical machine learning and is beginning to be used in critical applied settings, it is important to assess systematically how well they respond to the kind of challenges their clas…
Do physics-informed neural networks (PINNs) need to be deep? Shallow PINNs using the Levenberg-Marquardt algorithm
organization= Department of Mathematics, Institut Teknologi Sepuluh Nopember , city= Surabaya , postcode= 60111 , country= Indonesia；organization= Department of Mathematics, Khalifa University of Science \& Technology , city= Abu Dhabi , postcode= PO Box 127788 , country= United Arab Emirates；organization= Department of Mathematics, Faculty of Mathematics
Indraprastha Institute of Information Technology, Delhi（因德拉普拉斯信息技术学院（德里））
The concept of deep dictionary learning has been recently proposed. Unlike shallow dictionary learning which learns single level of dictionary to represent the data, it uses multiple layers of dictionaries. So far, the problem could only be solved in a greedy fashion; this was achieved by learning a single layer of dictionary in each stage where the coefficients from the previous layer acted as inputs to the subsequent layer (only the first layer used the training samples as inputs). This was not optimal; there was feedback from shallower to deeper layers but not the other way. This work propo…
University of Trento（特伦托大学）；North Carolina State University（北卡罗来纳州立大学）
Millimeter wave (mmWave) localization algorithms exploit the quasi-optical propagation of mmWave signals, which yields sparse angular spectra at the receiver. Geometric approaches to angle-based localization typically require to know the map of the environment and the location of the access points. Thus, several works have resorted to automated learning in order to infer a device's location from the properties of the received mmWave signals. However, collecting training data for such models is a significant burden. In this work, we propose a shallow neural network model to localize mmWave devi…
Why "classic" Transformers are shallow and how to make them go deep
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
