Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as well as the prediction accuracy. Coarse-grained pruning creates regular sparsity patterns, making it more amenable for hardware acceleration but more challenging to maintain the same accuracy. In this paper we quantitatively measure the trade-off between sparsity regularity and prediction accuracy, providing insights in how to maintain accuracy while having more a more structured sparsity pattern. Our experimental results show that coarse-grained pruning can achieve a sparsity ratio similar to unstructured pruning without loss of accuracy. Moreover, due to the index saving effect, coarse-grained pruning is able to obtain a better compression ratio than fine-grained sparsity at the same accuracy threshold. Based on the recent sparse convolutional neural network accelerator (SCNN), our experiments further demonstrate that coarse-grained sparsity saves about 2x the memory references compared to fine-grained sparsity. Since memory reference is more than two orders of magnitude more expensive than arithmetic operations, the regularity of sparse structure leads to more efficient hardware design.
Preferred Networks(优选网络公司);The University of Tokyo(东京大学)
Tensor decomposition methods are widely used for model compression and fast inference in convolutional neural networks (CNNs). Although many decompositions are conceivable, only CP decomposition and a few others have been applied in practice, and no extensive comparisons have been made between available methods. Previous studies have not determined how many decompositions are available, nor which of them is optimal. In this study, we first characterize a decomposition class specific to CNNs by adopting a flexible graphical notation. The class includes such well-known CNN modules as depthwise s…
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology (BNRist)(北京国家信息科学与技术研究中心(BNRist))
In this work, we propose a low-bit training framework for convolutional neural networks, which is built around a novel multi-level scaling (MLS) tensor format. Our framework focuses on reducing the energy consumption of convolution operations by quantizing all the convolution operands to low bit-width format. Specifically, we propose the MLS tensor format, in which the element-wise bit-width can be largely reduced. Then, we describe the dynamic quantization and the low-bit tensor convolution arithmetic to leverage the MLS tensor format efficiently. Experiments show that our framework achieves …
In recent years, the research community has discovered that deep neural networks (DNNs) and convolutional neural networks (CNNs) can yield higher accuracy than all previous solutions to a broad array of machine learning problems. To our knowledge, there is no single CNN/DNN architecture that solves all problems optimally. Instead, the "right" CNN/DNN architecture varies depending on the application at hand. CNN/DNNs comprise an enormous design space. Quantitatively, we find that a small region of the CNN design space contains 30 billion different CNN architectures. In this dissertation, we dev…
The past few years have witnessed growth in the computational requirements for training deep convolutional neural networks. Current approaches parallelize training onto multiple devices by applying a single parallelization strategy (e.g., data or model parallelism) to all layers in a network. Although easy to reason about, these approaches result in suboptimal runtime performance in large-scale distributed training, since different layers in a network may prefer different parallelization strategies. In this paper, we propose layer-wise parallelism that allows each layer in a network to use an …
University of Toronto(多伦多大学);Vector Institute(向量研究所)
Triangular, overlapping Mel-scaled filters ("f-banks") are the current standard input for acoustic models that exploit their input's time-frequency geometry, because they provide a psycho-acoustically motivated time-frequency geometry for a speech signal. F-bank coefficients are provably robust to small deformations in the scale. In this paper, we explore two ways in which filter banks can be adjusted for the purposes of speech recognition. First, triangular filters can be replaced with Gabor filters, a compactly supported filter that better localizes events in time, or Gammatone filters, a ps…
This paper aims at studying the sample complexity of graph convolutional networks (GCNs), by providing tight upper bounds of Rademacher complexity for GCN models with a single hidden layer. Under regularity conditions, theses derived complexity bounds explicitly depend on the largest eigenvalue of graph convolution filter and the degree distribution of the graph. Again, we provide a lower bound of Rademacher complexity for GCNs to show optimality of our derived upper bounds. Taking two commonly used examples as representatives, we discuss the implications of our results in designing graph conv…
This paper tackles the problem of Lipschitz regularization of Convolutional Neural Networks. Lipschitz regularity is now established as a key property of modern deep learning with implications in training stability, generalization, robustness against adversarial examples, etc. However, computing the exact value of the Lipschitz constant of a neural network is known to be NP-hard. Recent attempts from the literature introduce upper bounds to approximate this constant that are either efficient but loose or accurate but computationally expensive. In this work, by leveraging the theory of Toeplitz…
A counter-intuitive property of convolutional neural networks (CNNs) is their inherent susceptibility to adversarial examples, which severely hinders the application of CNNs in security-critical fields. Adversarial examples are similar to original examples but contain malicious perturbations. Adversarial training is a simple and effective defense method to improve the robustness of CNNs to adversarial examples. The mechanisms behind adversarial examples and adversarial training are worth exploring. Therefore, this work investigates similarities and differences between normally trained CNNs (NT…
Understanding the features learned by deep models is important from a model trust perspective, especially as deep systems are deployed in the real world. Most recent approaches for deep feature understanding or model explanation focus on highlighting input data features that are relevant for classification decisions. In this work, we instead take the perspective of relating deep features to well-studied, hand-crafted features that are meaningful for the application of interest. We propose a methodology and set of systematic experiments for exploring deep features in this setting, where input f…
Deep neural network is difficult to train and this predicament becomes worse as the depth increases. The essence of this problem exists in the magnitude of backpropagated errors that will result in gradient vanishing or exploding phenomenon. We show that a variant of regularizer which utilizes orthonormality among different filter banks can alleviate this problem. Moreover, we design a backward error modulation mechanism based on the quasi-isometry assumption between two consecutive parametric layers. Equipped with these two ingredients, we propose several novel optimization solutions that can…
School of Computing, Newcastle University(纽卡斯尔大学计算机学院);Department of Computer Science, Durham University(杜伦大学计算机科学系)
With the growing significance of graphs as an effective representation of data in numerous applications, efficient graph analysis using modern machine learning is receiving a growing level of attention. Deep learning approaches often operate over the entire adjacency matrix -- as the input and intermediate network layers are all designed in proportion to the size of the adjacency matrix -- leading to intensive computation and large memory requirements as the graph size increases. It is therefore desirable to identify efficient measures to reduce both run-time and memory requirements allowing f…
Asia University(亚洲大学);China Medical University Hospital, China Medical University(中国医药大学附设医院,中国医药大学);Universitas Muhammadiyah Yogyakarta(穆罕默迪亚日惹大学);Western Norway University of Applied Sciences(挪威西部应用科学大学)
The spread of PM2.5 pollutants that endanger health is difficult to predict because it involves many atmospheric variables. These micron particles can spread rapidly from their source to residential areas, increasing the risk of respiratory disease if exposed for long periods. The prediction system of PM2.5 propagation provides more detailed and accurate information as an early warning system to reduce health impacts on the community. According to the idea of transformative computing, the approach we propose in this paper allows computation on the dataset obtained from massive-scale PM2.5 sens…
Exploring specialization and sensitivity of convolutional neural networks in the context of simultaneous image augmentations
Marchuk Institute of Numerical Mathematics(马尔库克数值数学研究所);Skolkovo Institute of Science and Technology(斯克尔科沃科学与技术研究所);Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University(洛蒙诺索夫莫斯科国立大学计算数学与 cybernetics 学院);Artificial Intelligence Research Institute (AIRI)(人工智能研究 institute (AIRI))
Technische Universität München(慕尼黑工业大学);Klinikum der Universität München(慕尼黑大学附属医院);Johns Hopkins University(约翰斯·霍普金斯大学)
Structural data from Electronic Health Records as complementary information to imaging data for disease prediction. We incorporate novel weighting layer into the Graph Convolutional Networks, which weights every element of structural data by exploring its relation to the underlying disease. We demonstrate the superiority of our developed technique in terms of computational speed and obtained encouraging results where our method outperforms the state-of-the-art methods when applied to two publicly available datasets ABIDE and Chest X-ray in terms of relative performance for the accuracy of pred…
There is a need for automatic diagnosis of certain diseases from medical images that could help medical practitioners for further assessment towards treating the illness. Alzheimers disease is a good example of a disease that is often misdiagnosed. Alzheimers disease (Hear after referred to as AD), is caused by atrophy of certain brain regions and by brain cell death and is the leading cause of dementia and memory loss [1]. MRI scans reveal this information but atrophied regions are different for different individuals which makes the diagnosis a bit more trickier and often gets misdiagnosed [1…
We introduce a novel convolutional network architecture with an interpretable latent space for modeling guitar amplifiers. Leveraging domain knowledge of popular amplifiers spanning a range of styles, the proposed system intuitively combines or subtracts characteristics of different amplifiers, allowing musicians to design entirely new guitar timbres.
Filters in a Convolutional Neural Network (CNN) contain model parameters learned from enormous amounts of data. In this paper, we suggest to decompose convolutional filters in CNN as a truncated expansion with pre-fixed bases, namely the Decomposed Convolutional Filters network (DCFNet), where the expansion coefficients remain learned from data. Such a structure not only reduces the number of trainable parameters and computation, but also imposes filter regularity by bases truncation. Through extensive experiments, we consistently observe that DCFNet maintains accuracy for image classification…
Exploring the Role of Convolutional Neural Networks (CNN) in Dental Radiography Segmentation: A Comprehensive Systematic Literature Review
Research Groups in Intelligent Machines (REGIM Lab), University of Sfax, National Engineering School of Sfax (ENIS)(智能机器研究组(REGIM实验室),斯法克斯大学,斯法克斯国家工程学院)
In this study, we proposed a convolutional neural network model for gender prediction using English Twitter text as input. Ensemble of proposed model achieved an accuracy at 0.8237 on gender prediction and compared favorably with the state-of-the-art performance in a recent author profiling task. We further leveraged the trained models to predict the gender labels from an HPV vaccine related corpus and identified gender difference in public perceptions regarding HPV vaccine. The findings are largely consistent with previous survey-based studies.
Previous models for learning entity and relationship embeddings of knowledge graphs such as TransE, TransH, and TransR aim to explore new links based on learned representations. However, these models interpret relationships as simple translations on entity embeddings. In this paper, we try to learn more complex connections between entities and relationships. In particular, we use a Convolutional Neural Network (CNN) to learn entity and relationship representations in knowledge graphs. In our model, we treat entities and relationships as one-dimensional numerical sequences with the same length.…
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所)
Graph Convolutional Neural Networks (GCNNs) are generalizations of CNNs to graph-structured data, in which convolution is guided by the graph topology. In many cases where graphs are unavailable, existing methods manually construct graphs or learn task-driven adaptive graphs. In this paper, we propose Graph Learning Neural Networks (GLNNs), which exploit the optimization of graphs (the adjacency matrix in particular) from both data and tasks. Leveraging on spectral graph theory, we propose the objective of graph learning from a sparsity constraint, properties of a valid adjacency matrix as wel…
Université Paris-Est(巴黎东部大学);École des Ponts ParisTech(巴黎高科桥路学校);Inria(法国国家信息与自动化研究所)
We propose to impose symmetry in neural network parameters to improve parameter usage and make use of dedicated convolution and matrix multiplication routines. Due to significant reduction in the number of parameters as a result of the symmetry constraints, one would expect a dramatic drop in accuracy. Surprisingly, we show that this is not the case, and, depending on network size, symmetry can have little or no negative effect on network accuracy, especially in deep overparameterized networks. We propose several ways to impose local symmetry in recurrent and convolutional neural networks, and…
When working with three-dimensional data, choice of representation is key. We explore voxel-based models, and present evidence for the viability of voxellated representations in applications including shape modeling and object classification. Our key contributions are methods for training voxel-based variational autoencoders, a user interface for exploring the latent space learned by the autoencoder, and a deep convolutional neural network architecture for object classification. We address challenges unique to voxel-based representations, and empirically evaluate our models on the ModelNet ben…
Artificial intelligence (AI) is playing an increasingly significant role in our everyday lives. This trend is expected to continue, especially with recent pushes to move more AI to the edge. However, one of the biggest challenges associated with AI on edge devices (mobile phones, unmanned vehicles, sensors, etc.) is their associated size, weight, and power constraints. In this work, we consider the scenario where an AI system may need to operate at less-than-maximum accuracy in order to meet application-dependent energy requirements. We propose a simple function that divides the cost of using …
Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学)
Graph convolutional networks (GCNs) have shown promising results in processing graph data by extracting structure-aware features. This gave rise to extensive work in geometric deep learning, focusing on designing network architectures that ensure neuron activations conform to regularity patterns within the input graph. However, in most cases the graph structure is only accounted for by considering the similarity of activations between adjacent nodes, which limits the capabilities of such methods to discriminate between nodes in a graph. Here, we propose to augment conventional GCNs with geomet…
Center for Energy-Efficient Computing and Applications, Peking University(北京大学高能效计算与应用中心);Houmo AI(后摩智能)
Network quantization is a powerful technique to compress convolutional neural networks. The quantization granularity determines how to share the scaling factors in weights, which affects the performance of network quantization. Most existing approaches share the scaling factors layerwisely or channelwisely for quantization of convolutional layers. Channelwise quantization and layerwise quantization have been widely used in various applications. However, other quantization granularities are rarely explored. In this paper, we will explore the sub-layerwise granularity that shares the scaling fac…
Beijing Normal University(北京师范大学);Hong Kong Baptist University(香港浸会大学)
We use Deep Convolutional Neural Networks (DCNNs) for image segmentation problems. DCNNs can well extract the features from natural images. However, the classification functions in the existing network architecture of CNNs are simple and lack capabilities to handle important spatial information in a way that have been done for many well-known traditional variational models. Prior such as spatial regularity, volume prior and object shapes cannot be well handled by existing DCNNs. We propose a novel Soft Threshold Dynamics (STD) framework which can easily integrate many spatial priors of the cla…
Neural networks have become the standard model for various computer vision tasks in automated driving including semantic segmentation, moving object detection, depth estimation, visual odometry, etc. The main flavors of neural networks which are used commonly are convolutional (CNN) and recurrent (RNN). In spite of rapid progress in embedded processors, power consumption and cost is still a bottleneck. Spiking Neural Networks (SNNs) are gradually progressing to achieve low-power event-driven hardware architecture which has a potential for high efficiency. In this paper, we explore the role of …
Binghamton University(宾汉姆顿大学);IBM T.J. Watson Research Center(IBM T.J.沃森研究中心);Tsinghua University(清华大学)
Building large models with parameter sharing accounts for most of the success of deep convolutional neural networks (CNNs). In this paper, we propose doubly convolutional neural networks (DCNNs), which significantly improve the performance of CNNs by further exploring this idea. In stead of allocating a set of convolutional filters that are independently learned, a DCNN maintains groups of filters where filters within each group are translated versions of each other. Practically, a DCNN can be easily implemented by a two-step convolution procedure, which is supported by most modern deep learni…
Université Clermont Auvergne(克莱蒙奥弗涅大学);CNRS(法国国家科学研究中心);ENSMSE(圣艾蒂安国立矿业工程师学校);INRAE(法国国家农业食品与环境研究院);VetAgro Sup(法国兽医与农学高等学院);Université de Lyon(里昂大学);CHU Clermont-Ferrand(克莱蒙费朗大学医院);Inserm(法国国家健康与医学研究院);Tous Chercheurs Laboratory(全民研究者实验室);CPIE Champenoux(尚佩努环境倡议中心);University Hospital of Nancy(南锡大学医院);University Hospital of Saint Etienne(圣艾蒂安大学医院);IHU-Méditerranée Infection(地中海感染大学医院研究所);Aix Marseille Univ(艾克斯-马赛大学);IRD(法国发展研究院);AP-HM(马赛公立医院援助机构);SSA(法国军队卫生勤务局);Hôpitaux Universitaires(斯特拉斯堡大学医院);Pontchaillou University Hospital(蓬查卢大学医院);CHR Orléans(奥尔良地区中心医院);Hospital of Villeneuve-Saint-Georges(维勒讷夫圣乔治医院);Centre Hospitalier Universitaire de Rennes(雷恩大学医院)
Lyme disease which is one of the most common infectious vector-borne diseases manifests itself in most cases with erythema migrans (EM) skin lesions. Recent studies show that convolutional neural networks (CNNs) perform well to identify skin lesions from images. Lightweight CNN based pre-scanner applications for resource-constrained mobile devices can help users with early diagnosis of Lyme disease and prevent the transition to a severe late form thanks to appropriate antibiotic therapy. Also, resource-intensive CNN based robust computer applications can assist non-expert practitioners with an…
Exploring Geometric Deep Learning For Precipitation Nowcasting
Technical University of Munich (TUM)(慕尼黑工业大学);Potsdam Institute for Climate Impact Research(波茨坦气候影响研究所);Indian Institute of Technology Delhi(印度理工学院德里分校)
In this paper, we present an experiment on using deep learning and transfer learning techniques for emotion analysis in tweets and suggest a method to interpret our deep learning models. The proposed approach for emotion analysis combines a Long Short Term Memory (LSTM) network with a Convolutional Neural Network (CNN). Then we extend this approach for emotion intensity prediction using transfer learning technique. Furthermore, we propose a technique to visualize the importance of each word in a tweet to get a better understanding of the model. Experimentally, we show in our analysis that the …
Exploring Randomly Wired Neural Networks for Climate Model Emulation
Harvey Mudd College(哈维穆德学院);University of Southern California(南加州大学);Pacific Northwest National Laboratory(太平洋西北国家实验室);University of California San Diego(加州大学圣迭戈分校)
University of Washington(华盛顿大学);Cleveland Clinic Foundation(克利夫兰诊所基金会);Google AI Healthcare(谷歌AI医疗)
Seizure detection from EEGs is a challenging and time consuming clinical problem that would benefit from the development of automated algorithms. EEGs can be viewed as structural time series, because they are multivariate time series where the placement of leads on a patient's scalp provides prior information about the structure of interactions. Commonly used deep learning models for time series don't offer a way to leverage structural information, but this would be desirable in a model for structural time series. To address this challenge, we propose the temporal graph convolutional network (…
Georgia Institute of Technology(佐治亚理工学院);University of Enna(恩纳大学);Tencent Corporation(腾讯公司)
This paper investigates different trade-offs between the number of model parameters and enhanced speech qualities by employing several deep tensor-to-vector regression models for speech enhancement. We find that a hybrid architecture, namely CNN-TT, is capable of maintaining a good quality performance with a reduced model parameter size. CNN-TT is composed of several convolutional layers at the bottom for feature extraction to improve speech quality and a tensor-train (TT) output layer on the top to reduce model parameters. We first derive a new upper bound on the generalization power of the c…
Visual question answering is a recently proposed artificial intelligence task that requires a deep understanding of both images and texts. In deep learning, images are typically modeled through convolutional neural networks, and texts are typically modeled through recurrent neural networks. While the requirement for modeling images is similar to traditional computer vision tasks, such as object recognition and image classification, visual question answering raises a different need for textual representation as compared to other natural language processing tasks. In this work, we perform a deta…
Graph Convolutional Networks (GCNs) have been proved successful in the field of semi-supervised node classification by extracting structural information from graph data. However, the random selection of labeled nodes used by GCNs may lead to unstable generalization performance of GCNs. In this paper, we propose an efficient method for the deterministic selection of labeled nodes: the Determinate Node Selection (DNS) algorithm. The DNS algorithm identifies two categories of representative nodes in the graph: typical nodes and divergent nodes. These labeled nodes are selected by exploring the st…
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院)
Both the Dictionary Learning (DL) and Convolutional Neural Networks (CNN) are powerful image representation learning systems based on different mechanisms and principles, however whether we can seamlessly integrate them to improve the per-formance is noteworthy exploring. To address this issue, we propose a novel generalized end-to-end representation learning architecture, dubbed Convolutional Dictionary Pair Learning Network (CDPL-Net) in this paper, which integrates the learning schemes of the CNN and dictionary pair learning into a unified framework. Generally, the architecture of CDPL-Net …
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);School of Cyberspace, Hangzhou Dianzi University(杭州电子科技大学网络空间安全学院)
Graph Convolutional Network (GCN) has shown remarkable potential of exploring graph representation. However, the GCN aggregating mechanism fails to generalize to networks with heterophily where most nodes have neighbors from different classes, which commonly exists in real-world networks. In order to make the propagation and aggregation mechanism of GCN suitable for both homophily and heterophily (or even their mixture), we introduce block modeling into the framework of GCN so that it can realize "block-guided classified aggregation", and automatically learn the corresponding aggregation rules…
The Hong Kong University of Science and Technology(香港科技大学);The Tongji University(同济大学)
Convolutional neural networks (CNNs) are widely used in many image recognition tasks due to their extraordinary performance. However, training a good CNN model can still be a challenging task. In a training process, a CNN model typically learns a large number of parameters over time, which usually results in different performance. Often, it is difficult to explore the relationships between the learned parameters and the model performance due to a large number of parameters and different random initializations. In this paper, we present a visual analytics approach to compare two different snaps…
Nagoya University(名古屋大学);Didi Chuxing(滴滴出行);Dalian University of Technology(大连理工大学)
Dynamic demand prediction is a key issue in ride-hailing dispatching. Many methods have been developed to improve the demand prediction accuracy of an increase in demand-responsive, ride-hailing transport services. However, the uncertainties in predicting ride-hailing demands due to multiscale spatiotemporal granularity, as well as the resulting statistical errors, are seldom explored. This paper attempts to fill this gap and to examine the spatiotemporal granularity effects on ride-hailing demand prediction accuracy by using empirical data for Chengdu, China. A convolutional, long short-term …
Shanghai Jiao Tong University(上海交通大学);University of California, Los Angeles(加州大学洛杉矶分校);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
A collection of approaches based on graph convolutional networks have proven success in skeleton-based action recognition by exploring neighborhood information and dense dependencies between intra-frame joints. However, these approaches usually ignore the spatial-temporal global context as well as the local relation between inter-frame and intra-frame. In this paper, we propose a focusing and diffusion mechanism to enhance graph convolutional networks by paying attention to the kinematic dependence of articulated human pose in a frame and their implicit dependencies over frames. In the focusin…
Exploring Various Sequential Learning Methods for Deformation History Modeling
Tallinn University of Technology(塔林理工大学);Aalto University(阿尔托大学);Nasa Research Glenn Center(美国国家航空航天局格伦研究中心);Bogazici University(博阿齐奇大学);College of Science and Engineering, Hamad Bin Khalifa University(哈马德·本·哈利法大学科学与工程学院)
Beijing Key Laboratory of Security and Privacy in Intelligent Transportation, Beijing Jiaotong University(北京交通大学智能交通信息安全与隐私保护北京市重点实验室)
In recent years, malware becomes more threatening. Concerning the increasing malware variants, there comes Machine Learning (ML)-based and Deep Learning (DL)-based approaches for heuristic detection. Nevertheless, the prediction accuracy of both needs to be improved. In response to the above issues in the PE malware domain, we propose the DL-based approaches for detection and use static-based features fed up into models. The contributions are as follows: we recapitulate existing malware detection methods. That is, we propose a vec-torized representation model of the malware instruction layer a…
University of South Carolina(南卡罗来纳大学);Guizhou University(贵州大学)
Machine learning (ML) methods have gained increasing popularity in exploring and developing new materials. More specifically, graph neural network (GNN) has been applied in predicting material properties. In this work, we develop a novel model, GATGNN, for predicting inorganic material properties based on graph neural networks composed of multiple graph-attention layers (GAT) and a global attention layer. Through the application of the GAT layers, our model can efficiently learn the complex bonds shared among the atoms within each atom's local neighborhood. Subsequently, the global attention l…
City University of Hong Kong(香港城市大学);Baidu USA LLC(百度美国有限责任公司)
Model pruning aims to reduce the deep neural network (DNN) model size or computational overhead. Traditional model pruning methods such as l-1 pruning that evaluates the channel significance for DNN pay too much attention to the local analysis of each channel and make use of the magnitude of the entire feature while ignoring its relevance to the batch normalization (BN) and ReLU layer after each convolutional operation. To overcome these problems, we propose a new model pruning method from a new perspective of gradient flow in this paper. Specifically, we first theoretically analyze the channe…