There has been a recent shift in sequence-to-sequence modeling from recurrent network architectures to convolutional network architectures due to computational advantages in training and operation while still achieving competitive performance. For systems having limited long-term temporal dependencies, the approximation capability of recurrent networks is essentially equivalent to that of temporal convolutional nets (TCNs). We prove that TCNs can approximate a large class of input-output maps having approximately finite memory to arbitrary error tolerance. Furthermore, we derive quantitative approximation rates for deep ReLU TCNs in terms of the width and depth of the network and modulus of continuity of the original input-output map, and apply these results to input-output maps of systems that admit finite-dimensional state-space realizations (i.e., recurrent models).
Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
UC Irvine（加州大学欧文分校）；University of Edinburgh（爱丁堡大学）；Ecole Polytechnique de Montreal（蒙特利尔理工学院）；University of Alberta（阿尔伯塔大学）；University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
Convolutional neural networks often dominate fully-connected counterparts in generalization performance, especially on image classification tasks. This is often explained in terms of 'better inductive bias'. However, this has not been made mathematically rigorous, and the hurdle is that the fully connected net can always simulate the convolutional net (for a fixed task). Thus the training algorithm plays a role. The current work describes a natural task on which a provable sample complexity gap can be shown, for standard training algorithms. We construct a single natural distribution on $\math…
Institute of Physics, University of Amsterdam（阿姆斯特丹大学物理研究所）
We introduce a machine learning model, the q-CNN model, sharing key features with convolutional neural networks and admitting a tensor network description. As examples, we apply q-CNN to the MNIST and Fashion MNIST classification tasks. We explain how the network associates a quantum state to each classification label, and study the entanglement structure of these network states. In both our experiments on the MNIST and Fashion-MNIST datasets, we observe a distinct increase in both the left/right as well as the up/down bipartition entanglement entropy during training as the network learns the …
Temple University（天普大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
Nowadays, it is broadly recognized in the power system community that to meet the ever expanding energy sector's needs, it is no longer possible to rely solely on physics-based models and that reliable, timely and sustainable operation of energy systems is impossible without systematic integration of artificial intelligence (AI) tools. Nevertheless, the adoption of AI in power systems is still limited, while integration of AI particularly into distribution grid investment planning is still an uncharted territory. We make the first step forward to bridge this gap by showing how graph convolutio…
EU-Nets: Enhanced, Explainable and Parsimonious U-Nets
School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）
In this paper we review the mathematical foundations of convolutional neural nets (CNNs) with the goals of: i) highlighting connections with techniques from statistics, signal processing, linear algebra, differential equations, and optimization, ii) demystifying underlying computations, and iii) identifying new types of applications. CNNs are powerful machine learning models that highlight features from grid data to make predictions (regression and classification). The grid data object can be represented as vectors (in 1D), matrices (in 2D), or tensors (in 3D or higher dimensions) and can inco…
NSF Center for Cloud and Autonomic Computing（NSF云与自主计算中心）；Department of Computer Science（计算机科学系）；The University of Arizona（亚利桑那大学）；The University of North Texas（北得克萨斯大学）
Anomaly detection is critically important for intelligent surveillance systems to detect in a timely manner any malicious activities. Many video anomaly detection approaches using deep learning methods focus on a single camera video stream with a fixed scenario. These deep learning methods use large-scale training data with large complexity. As a solution, in this paper, we show how to use pre-trained convolutional neural net models to perform feature extraction and context mining, and then use denoising autoencoder with relatively low model complexity to provide efficient and accurate surveil…
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Univ. of California, Los Angeles（加州大学洛杉矶分校）；Google Inc.（谷歌公司）；CentraleSupélec（巴黎中央理工-高等电力学院）；INRIA（法国国家信息与自动化研究所）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
KTH Royal Institute of Technology（瑞典皇家理工学院）；MessageBird（MessageBird公司）；University of Washington（华盛顿大学）
Sum-Product Networks (SPNs) are hierarchical, graphical models that combine benefits of deep learning and probabilistic modeling. SPNs offer unique advantages to applications demanding exact probabilistic inference over high-dimensional, noisy inputs. Yet, compared to convolutional neural nets, they struggle with capturing complex spatial relationships in image data. To alleviate this issue, we introduce Deep Generalized Convolutional Sum-Product Networks (DGC-SPNs), which encode spatial features in a way similar to CNNs, while preserving the validity of the probabilistic SPN model. As opposed…
School of Computing Science（计算科学学院）；University of Glasgow（格拉斯哥大学）；OxBrdgRbtx Ltd（OxBrdgRbtx有限公司）
We demonstrate the feasibility of framing a classically learned deep neural network as an energy based model that can be processed on a one-step quantum annealer in order to exploit fast sampling times. We propose approaches to overcome two hurdles for high resolution image classification on a quantum processing unit (QPU): the required number and binary nature of the model states. With this novel method we successfully transfer a convolutional neural network to the QPU and show the potential for classification speedup of at least one order of magnitude.
On the Behavior of Convolutional Nets for Feature Extraction
Barcelona Supercomputing Center (BSC)（巴塞罗那超级计算中心（BSC））；Universitat Politècnica de Catalunya - BarcelonaTech (UPC)（加泰罗尼亚理工大学 - 巴塞罗那技术大学（UPC））；IBM T.J. Watson Research Center（IBM T.J. 沃森研究中心）
Saratoga High School（萨拉托加高中）；BRAC University JPG School of Public Health（BRAC大学JPG公共卫生学院）
Diabetes is one of the most prevalent chronic diseases in Bangladesh, and as a result, Diabetic Retinopathy (DR) is widespread in the population. DR, an eye illness caused by diabetes, can lead to blindness if it is not identified and treated in its early stages. Unfortunately, diagnosis of DR requires medically trained professionals, but Bangladesh has limited specialists in comparison to its population. Moreover, the screening process is often expensive, prohibiting many from receiving timely and proper diagnosis. To address the problem, we introduce a deep learning algorithm which screens f…
University of Notre Dame（圣母大学）；Lawrence Livermore National Laboratories（劳伦斯利弗莫尔国家实验室）
Imputation of missing data is a task that plays a vital role in a number of engineering and science applications. Often such missing data arise in experimental observations from limitations of sensors or post-processing transformation errors. Other times they arise from numerical and algorithmic constraints in computer simulations. One such instance and the application emphasis of this paper are numerical simulations of storm surge. The simulation data corresponds to time-series surge predictions over a number of save points within the geographic domain of interest, creating a spatio-temporal …
North Carolina State University（北卡罗来纳州立大学）；Pacific Northwest National Lab（太平洋西北国家实验室）；Texas A&M（德克萨斯农工大学）
This paper presents a novel, automated, generative adversarial networks (GAN) based synthetic feeder generation mechanism, abbreviated as FeederGAN. FeederGAN digests real feeder models represented by directed graphs via a deep learning framework powered by GAN and graph convolutional networks (GCN). Information of a distribution feeder circuit is extracted from its model input files so that the device connectivity is mapped onto the adjacency matrix and the device characteristics, such as circuit types (i.e., 3-phase, 2-phase, and 1-phase) and component attributes (e.g., length and current ra…
Beyond the generally deployed features for microstructure property prediction this study aims to improve the machine learned prediction by developing novel feature descriptors. Therefore, Bayesian infused data mining is conducted to acquire samples containing characteristics inexplicable to the current feature set, and suitable feature descriptors to describe these characteristics are proposed. The iterative development of feature descriptors resulted in 37 novel features, being able to reduce the prediction error by roughly one third. To further improve the predictive model, convolutional neu…
Parallel Layer Normalization for Universal Approximation
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE)（复杂与关键软件环境国家重点实验室）；Beihang University（北京航空航天大学）；Hangzhou International Innovation Institute（杭州国际创新研究院）
Accurate and timely estimation of precipitation is critical for issuing hazard warnings (e.g., for flash floods or landslides). Current remotely sensed precipitation products have a few hours of latency, associated with the acquisition and processing of satellite data. By applying a robust nowcasting system to these products, it is (in principle) possible to reduce this latency and improve their applicability, value, and impact. However, the development of such a system is complicated by the chaotic nature of the atmosphere, and the consequent rapid changes that can occur in the structures of …
Xi’an Jiaotong University（西安交通大学）；City University of Hong Kong（香港城市大学）
Compared with avid research activities of deep convolutional neural networks (DCNNs) in practice, the study of theoretical behaviors of DCNNs lags heavily behind. In particular, the universal consistency of DCNNs remains open. In this paper, we prove that implementing empirical risk minimization on DCNNs with expansive convolution (with zero-padding) is strongly universally consistent. Motivated by the universal consistency, we conduct a series of experiments to show that without any fully connected layers, DCNNs with expansive convolution perform not worse than the widely used deep neural net…
Nanjing University of Information Science & Technology（南京信息工程大学）
Cerebrovascular accident, or commonly known as stroke, is an acute disease with extreme impact on patients and healthcare systems and is the second largest cause of death worldwide. Fast and precise stroke lesion detection and location is an extreme important process with regards to stroke diagnosis, treatment, and prognosis. Except from the manual segmentation approach, machine learning based segmentation methods are the most promising ones when considering efficiency and accuracy, and convolutional neural network based models are the first of its kind. However, most of these neural network m…
