Neural network forms the foundation of deep learning and numerous AI applications. Classical neural networks are fully connected, expensive to train and prone to overfitting. Sparse networks tend to have convoluted structure search, suboptimal performance and limited usage. We proposed the novel uniform sparse network (USN) with even and sparse connectivity within each layer. USN has one striking property that its performance is independent of the substantial topology variation and enormous model space, thus offers a search-free solution to all above mentioned issues of neural networks. USN consistently and substantially outperforms the state-of-the-art sparse network models in prediction accuracy, speed and robustness. It even achieves higher prediction accuracy than the fully connected network with only 0.55% parameters and 1/4 computing time and resources. Importantly, USN is conceptually simple as a natural generalization of fully connected network with multiple improvements in accuracy, robustness and scalability. USN can replace the latter in a range of applications, data types and deep learning architectures. We have made USN open source at https://github.com/datapplab/sparsenet.
Harvard University（哈佛大学）；School of Engineering and Applied Sciences (SEAS)（工程与应用科学学院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
We propose a novel denoising framework for task functional Magnetic Resonance Imaging (tfMRI) data to delineate the high-resolution spatial pattern of the brain functional connectivity via dictionary learning and sparse coding (DLSC). In order to address the limitations of the unsupervised DLSC-based fMRI studies, we utilize the prior knowledge of task paradigm in the learning step to train a data-driven dictionary and to model the sparse representation. We apply the proposed DLSC-based method to Human Connectome Project (HCP) motor tfMRI dataset. Studies on the functional connectivity of cere…
Subspace clustering is the problem of clustering data points into a union of low-dimensional linear/affine subspaces. It is the mathematical abstraction of many important problems in computer vision, image processing and machine learning. A line of recent work (4, 19, 24, 20) provided strong theoretical guarantee for sparse subspace clustering (4), the state-of-the-art algorithm for subspace clustering, on both noiseless and noisy data sets. It was shown that under mild conditions, with high probability no two points from different subspaces are clustered together. Such guarantee, however, is …
Division of Biostatistics, University of California, Berkeley（加州大学伯克利分校生物统计学系）
Tensor regression models, such as CP regression and Tucker regression, have many successful applications in neuroimaging analysis where the covariates are of ultrahigh dimensionality and possess complex spatial structures. The high-dimensional covariate arrays, also known as tensors, can be approximated by low-rank structures and fit into the generalized linear models. The resulting tensor regression achieves a significant reduction in dimensionality while remaining efficient in estimation and prediction. Brain functional connectivity is an essential measure of brain activity and has shown sig…
We provide novel guaranteed approaches for training feedforward neural networks with sparse connectivity. We leverage on the techniques developed previously for learning linear networks and show that they can also be effectively adopted to learn non-linear networks. We operate on the moments involving label and the score function of the input, and show that their factorization provably yields the weight matrix of the first layer of a deep network under mild conditions. In practice, the output of our method can be employed as effective initializers for gradient descent.
Recurrent neural networks (RNNs) are omnipresent in sequence modeling tasks. Practical models usually consist of several layers of hundreds or thousands of neurons which are fully connected. This places a heavy computational and memory burden on hardware, restricting adoption in practical low-cost and low-power devices. Compared to fully convolutional models, the costly sequential operation of RNNs severely hinders performance on parallel hardware. This paper challenges the convention of full connectivity in RNNs. We study structurally sparse RNNs, showing that they are well suited for acceler…
driveU（driveU公司）；Institute of Measurement, Control and Microtechnology（测量、控制与微技术研究所）；Ulm University（乌尔姆大学）；Institute of Neural Information Processing（神经信息处理研究所）
Sparseness is a useful regularizer for learning in a wide range of applications, in particular in neural networks. This paper proposes a model targeted at classification tasks, where sparse activity and sparse connectivity are used to enhance classification capabilities. The tool for achieving this is a sparseness-enforcing projection operator which finds the closest vector with a pre-defined sparseness for any given vector. In the theoretical part of this paper, a comprehensive theory for such a projection is developed. In conclusion, it is shown that the projection is differentiable almost e…
Large neural networks are very successful in various tasks. However, with limited data, the generalization capabilities of deep neural networks are also very limited. In this paper, we empirically start showing that intrinsically sparse neural networks with adaptive sparse connectivity, which by design have a strict parameter budget during the training phase, have better generalization capabilities than their fully-connected counterparts. Besides this, we propose a new technique to train these sparse models by combining the Sparse Evolutionary Training (SET) procedure with neurons pruning. Ope…
EEG-SeeGraph: Interpreting functional connectivity disruptions in dementias via sparse-explanatory dynamic EEG-graph learning
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Institute of Neurological Diseases, Shenzhen Bay Laboratory, Shenzhen, China（深圳湾实验室神经疾病研究院）
The over-parametrized nature of Deep Neural Networks leads to considerable hindrances during deployment on low-end devices with time and space constraints. Network pruning strategies that sparsify DNNs using iterative prune-train schemes are often computationally expensive. As a result, techniques that prune at initialization, prior to training, have become increasingly popular. In this work, we propose neuron-to-neuron skip connections, which act as sparse weighted skip connections, to enhance the overall connectivity of pruned DNNs. Following a preliminary pruning step, N2NSkip connections a…
Adversarially Robust Spiking Neural Networks with Sparse Connectivity
Institute of Machine Learning and Neural Computation, Graz University of Technology, Austria（机器学习与神经计算研究所，格拉茨技术大学）；TU Graz - SAL Dependable Embedded Systems Lab, Silicon Austria Labs, Austria（格拉茨大学 - SAL 可靠嵌入式系统实验室，Silicon Austria Labs）；Chair of Cyber-Physical-Systems, Montanuniversität Leoben, Austria（智能物理系统系，蒙特安大学）
Xiamen University（厦门大学）；Southern Illinois University Carbondale（南伊利诺伊大学卡本代尔分校）；Kneron Inc.（耐能智慧股份有限公司）；Yulin Normal University（玉林师范学院）
Since sparse neural networks usually contain many zero weights, these unnecessary network connections can potentially be eliminated without degrading network performance. Therefore, well-designed sparse neural networks have the potential to significantly reduce FLOPs and computational resources. In this work, we propose a new automatic pruning method - Sparse Connectivity Learning (SCL). Specifically, a weight is re-parameterized as an element-wise multiplication of a trainable weight variable and a binary mask. Thus, network connectivity is fully described by the binary mask, which is modulat…
Eindhoven University of Technology（埃因霍温理工大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of Twente（特文特大学）；Scuola Superiore Sant’Anna（圣安娜高等研究院）
Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from scratch by optimizing the sparse structure during training. However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions. This comparison becomes increasingly complex as the number of possi…
Advanced Communications Research Institute (ACRI)（高级通信研究所（ACRI））；Sharif University of Technology（谢里夫理工大学）
Sparse Inverse Covariance Estimation (SICE) is useful in many practical data analyses. Recovering the connectivity, non-connectivity graph of covariates is classified amongst the most important data mining and learning problems. In this paper, we introduce a novel SICE approach using adaptive thresholding. Our method is based on updates in a transformed domain of the desired matrix and exponentially decaying adaptive thresholding in the main domain (Inverse Covariance matrix domain). In addition to the proposed algorithm, the convergence analysis is also provided. In the Numerical Experiments …
We propose a novel way of reducing the number of parameters in the storage-hungry fully connected layers of a neural network by using pre-defined sparsity, where the majority of connections are absent prior to starting training. Our results indicate that convolutional neural networks can operate without any loss of accuracy at less than half percent classification layer connection density, or less than 5 percent overall network connection density. We also investigate the effects of pre-defining the sparsity of networks with only fully connected layers. Based on our sparsifying technique, we in…
Eindhoven University of Technology（埃因霍温理工大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Derby（德比大学）
Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős-Rényi random graph) of two consecutive layers of neurons into a scale-free topology,…
Deep neural networks have achieved human-level accuracy on almost all perceptual benchmarks. It is interesting that these advances were made using two ideas that are decades old: (a) an artificial neuron based on a linear summator and (b) SGD training. However, there are important metrics beyond accuracy: computational efficiency and stability against adversarial perturbations. In this paper, we propose two closely connected methods to improve these metrics on contour recognition tasks: (a) a novel model of an artificial neuron, a "strong neuron," with low hardware requirements and inherent ro…
On Improving Graph Neural Networks for QSAR by Pre-training on Extended-Connectivity Fingerprints
Department of Statistics, University of Oxford（牛津大学统计系）；Mathematical Institute, University of Oxford（牛津大学数学院）；Molecular Informatics and AI, Lhasa Limited（Lhasa有限公司分子信息学与人工智能部门）
Center for Security Theory and Algorithmic Research（安全理论与算法研究中心）；International Institute of Information Technology Hyderabad（海得拉巴国际信息技术学院）
Sparse neural networks are shown to give accurate predictions competitive to denser versions, while also minimizing the number of arithmetic operations performed. However current hardware like GPU's can only exploit structured sparsity patterns for better efficiency. Hence the run time of a sparse neural network may not correspond to the arithmetic operations required. In this work, we propose RBGP( Ramanujan Bipartite Graph Product) framework for generating structured multi level block sparse neural networks by using the theory of Graph products. We also propose to use products of Ramanujan g…
Mode connectivity provides novel geometric insights on analyzing loss landscapes and enables building high-accuracy pathways between well-trained neural networks. In this work, we propose to employ mode connectivity in loss landscapes to study the adversarial robustness of deep neural networks, and provide novel methods for improving this robustness. Our experiments cover various types of adversarial attacks applied to different network architectures and datasets. When network models are tampered with backdoor or error-injection attacks, our results demonstrate that the path connection learned…
University of Toronto（多伦多大学）；Modiface, Inc.（Modiface公司）；University of Guelph（圭尔夫大学）；Vector Institute（向量研究所）
Vision Transformers (ViT) have shown their competitive advantages performance-wise compared to convolutional neural networks (CNNs) though they often come with high computational costs. To this end, previous methods explore different attention patterns by limiting a fixed number of spatially nearby tokens to accelerate the ViT's multi-head self-attention (MHSA) operations. However, such structured attention patterns limit the token-to-token connections to their spatial relevance, which disregards learned semantic connections from a full attention mask. In this work, we propose a novel approach…
The Hong Kong University of Science and Technology（香港科技大学）
Sparse connectivity is an important factor behind the success of convolutional neural networks and recurrent neural networks. In this paper, we consider the problem of learning sparse connectivity for feedforward neural networks (FNNs). The key idea is that a unit should be connected to a small number of units at the next level below that are strongly correlated. We use Chow-Liu's algorithm to learn a tree-structured probabilistic model for the units at the current level, use the tree to identify subsets of units that are strongly correlated, and introduce a new unit with receptive field over …
Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS)（神经影像与数据科学转化研究三机构中心（TReNDS））；Georgia State University（佐治亚州立大学）；Georgia Tech（佐治亚理工学院）；Emory（埃默里大学）
Functional connectivity (FC) studies have demonstrated the overarching value of studying the brain and its disorders through the undirected weighted graph of fMRI correlation matrix. Most of the work with the FC, however, depends on the way the connectivity is computed, and further depends on the manual post-hoc analysis of the FC matrices. In this work we propose a deep learning architecture BrainGNN that learns the connectivity structure as part of learning to classify subjects. It simultaneously applies a graphical neural network to this learned graph and learns to select a sparse subset of…
Traditional semantic similarity models often fail to encapsulate the external context in which texts are situated. However, textual datasets generated on mobile platforms can help us build a truer representation of semantic similarity by introducing multimodal data. This is especially important in sparse datasets, making solely text-driven interpretation of context more difficult. In this paper, we develop new algorithms for building external features into sentence embeddings and semantic similarity scores. Then, we test them on embedding spaces on data from Twitter, using each tweet's time an…
Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected
Center for Complex Network Intelligence (CCNI)（复杂网络智能研究中心）；Dept. of Computer Science & Technology（计算机科学与技术系）；School of Biomedical Engineering（生物医学工程学院）；University of Padova（帕多瓦大学）；Canva Research（Canva研究）
Sparse Equation Matching: A Derivative-Free Learning for General-Order Dynamical Systems
School of Management University of Science and Technology of China（科学技术大学管理学院）；Department of Biostatistics and Bioinformatics Duke University（杜克大学生物统计与生物信息学系）
Claremont Graduate University（克莱蒙特研究生大学）；Naval Air Warfare Center（海军空战中心）；Claremont McKenna College（克莱蒙特·麦肯纳学院）
We propose a method to improve image clustering using sparse text and the wisdom of the crowds. In particular, we present a method to fuse two different kinds of document features, image and text features, and use a common dictionary or "wisdom of the crowds" as the connection between the two different kinds of documents. With the proposed fusion matrix, we use topic modeling via non-negative matrix factorization to cluster documents.
This paper considers joint learning of multiple sparse Granger graphical models to discover underlying common and differential Granger causality (GC) structures across multiple time series. This can be applied to drawing group-level brain connectivity inferences from a homogeneous group of subjects or discovering network differences among groups of signals collected under heterogeneous conditions. By recognizing that the GC of a single multivariate time series can be characterized by common zeros of vector autoregressive (VAR) lag coefficients, a group sparse prior is included in joint regular…
The study of healthy brain development helps to better understand the brain transformation and brain connectivity patterns which happen during childhood to adulthood. This study presents a sparse machine learning solution across whole-brain functional connectivity (FC) measures of three sets of data, derived from resting state functional magnetic resonance imaging (rs-fMRI) and task fMRI data, including a working memory n-back task (nb-fMRI) and an emotion identification task (em-fMRI). These multi-modal image data are collected on a sample of adolescents from the Philadelphia Neurodevelopment…
Imperial College London（帝国理工学院）；Technical University of Munich（慕尼黑工业大学）
The study of functional brain connectivity (FC) is important for understanding the underlying mechanisms of many psychiatric disorders. Many recent analyses adopt graph convolutional networks, to study non-linear interactions between functionally-correlated states. However, although patterns of brain activation are known to be hierarchically organised in both space and time, many methods have failed to extract powerful spatio-temporal features. To overcome those challenges, and improve understanding of long-range functional dynamics, we translate an approach, from the domain of skeleton-based …
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total …
Neuro-inspired Ensemble-to-Ensemble Communication Primitives for Sparse and Efficient ANNs
Archimedes, Athena Research Center（阿基米德研究中心）；Harvard Medical School（哈佛医学院）；University of Crete（希腊克里特大学）；Foundation for Research and Technology-Hellas（希腊研究与技术基金会）
Sparse Network Inference under Imperfect Detection and its Application to Ecological Networks
Department of Electrical and Computer Engineering and the Ken Kennedy Institute（电气与计算机工程系和肯尼迪研究所）；Department of Computational and Applied Mathematics at Rice University（里士满大学计算与应用数学系）；SISTEMIC, Facultad de Ingeniería, Universidad de Antioquia（安提奥基亚大学工程学院SISTEMIC）
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）；Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS)（三校神经影像与数据科学转化研究中心（TReNDS））；Georgia State University（佐治亚州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）
Independent component analysis (ICA) of multi-subject functional magnetic resonance imaging (fMRI) data has proven useful in providing a fully multivariate summary that can be used for multiple purposes. ICA can identify patterns that can discriminate between healthy controls (HC) and patients with various mental disorders such as schizophrenia (Sz). Temporal functional network connectivity (tFNC) obtained from ICA can effectively explain the interactions between brain networks. On the other hand, dictionary learning (DL) enables the discovery of hidden information in data using learnable basi…
Decentralized Sparse Linear Regression via Gradient-Tracking: Linear Convergence and Statistical Guarantees
Texas A&M University（德克萨斯农工大学）；Purdue University（普渡大学）；The Pennsylvania State University（宾夕法尼亚州立大学）；The University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Improving Clinical Dataset Condensation with Mode Connectivity-based Trajectory Surrogates
Institute of Biomedical Engineering, University of Oxford, UK（生物医学工程研究所，牛津大学，英国）；GlaxoSmithKline, London, UK（葛兰素史克，伦敦，英国）；Nuffield Department of Primary Care Health Sciences, University of Oxford, UK（初级保健健康科学尼弗尔部门，牛津大学，英国）；Oxford-Suzhou Institute of Advanced Research (OSCAR), Suzhou, China（牛津-苏州先进研究所（OSCAR），苏州，中国）
