arXiv 1606.024072016-06-08cs.NEcs.AIcs.CVcs.LG

Structured Convolution Matrices for Energy-efficient Deep learning

IBM Research–Almaden（IBM阿尔马登研究中心）

Rathinakumar Appuswamy, Tapan Nayak, John Arthur, Steven Esser, Paul Merolla, Jeffrey Mckinstry, Timothy Melano, Myron Flickner, Dharmendra Modha

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

We derive a relationship between network representation in energy-efficient neuromorphic architectures and block Toplitz convolutional matrices. Inspired by this connection, we develop deep convolutional networks using a family of structured convolutional matrices and achieve state-of-the-art trade-off between energy efficiency and classification accuracy for well-known image recognition tasks. We also put forward a novel method to train binary convolutional networks by utilising an existing connection between noisy-rectified linear units and binary activations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

用于高效深度神经网络的广义结构化矩阵可微学习

Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks

University of Michigan（密歇根大学）

针对高效DNN权重矩阵结构难确定、现有结构化矩阵多为手工设计的问题，提出通用可微框架，通过近端梯度下降学习广义结构化矩阵参数，在图像和语言任务上实现更优的复杂度与性能平衡。

2024-03-08cs.AIcs.CVcs.LG

A Generalization of Convolutional Neural Networks to Graph-Structured Data

Carnegie Mellon University（卡内基梅隆大学）

This paper introduces a generalization of Convolutional Neural Networks (CNNs) from low-dimensional grid data, such as images, to graph-structured data. We propose a novel spatial convolution utilizing a random walk to uncover the relations within the input, analogous to the way the standard convolution uses the spatial neighborhood of a pixel on the grid. The convolution has an intuitive interpretation, is efficient and scalable and can also be used on data with varying graph structure. Furthermore, this generalization can be applied to many standard regression or classification problems, by …

2017-04-26cs.AIcs.CVcs.LG

Lossless Compression of Structured Convolutional Models via Lifting

Czech Technical University in Prague（捷克布拉格技术大学）

Lifting is an efficient technique to scale up graphical models generalized to relational domains by exploiting the underlying symmetries. Concurrently, neural models are continuously expanding from grid-like tensor data into structured representations, such as various attributed graphs and relational databases. To address the irregular structure of the data, the models typically extrapolate on the idea of convolution, effectively introducing parameter sharing in their, dynamically unfolded, computation graphs. The computation graphs themselves then reflect the symmetries of the underlying data…

2021-01-18cs.AIcs.LG

BLAST：用于高效深度神经网络推理的块级自适应结构化矩阵

BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference

University of Michigan（密歇根大学）

针对大模型推理的密集矩阵运算计算挑战，提出BLAST块级自适应结构化矩阵，可灵活表示多种权重结构，在中/大模型上均实现高效压缩且性能损失更低。

2024-10-30cs.AIcs.LGstat.ML

Structured Convolutions for Efficient Neural Network Design

Qualcomm AI Research（高通人工智能研究院）

In this work, we tackle model efficiency by exploiting redundancy in the \textit{implicit structure} of the building blocks of convolutional neural networks. We start our analysis by introducing a general definition of Composite Kernel structures that enable the execution of convolution operations in the form of efficient, scaled, sum-pooling components. As its special case, we propose \textit{Structured Convolutions} and show that these allow decomposition of the convolution operation into a sum-pooling operation followed by a convolution with significantly lower complexity and fewer weights.…

2020-10-31cs.CVcs.LGcs.NE

通过掩码诱导半结构化稀疏性以实现卷积网络的高效模型推理

Inducing Semi-Structured Sparsity by Masking for Efficient Model Inference in Convolutional Networks

ETH Zürich（苏黎世联邦理工学院）

针对卷积模型的加速需求，提出通过掩码学习卷积核半结构化稀疏模式的方法，可在不降低性能、保留原模型权重结构的前提下实现两倍以上推理加速，且能推导掩码下预测的稳定性边界。

2024-11-01cs.AIcs.CVcs.LG

更多相关论文收起更多论文

Structured Sparse Convolutional Autoencoder

University of Louisville（路易斯维尔大学）

This paper aims to improve the feature learning in Convolutional Networks (Convnet) by capturing the structure of objects. A new sparsity function is imposed on the extracted featuremap to capture the structure and shape of the learned object, extracting interpretable features to improve the prediction performance. The proposed algorithm is based on organizing the activation within and across featuremap by constraining the node activities through $\ell_{2}$ and $\ell_{1}$ normalization in a structured form.

2017-01-02cs.LGcs.NE

Deep Convolutional Networks on Graph-Structured Data

Courant Institute of Mathematical Sciences（库朗数学科学研究所）；New York University（纽约大学）；University of California, Berkeley（加州大学伯克利分校）

Deep Learning's recent successes have mostly relied on Convolutional Networks, which exploit fundamental statistical properties of images, sounds and video data: the local stationarity and multi-scale compositional structure, that allows expressing long range interactions in terms of shorter, localized interactions. However, there exist other important examples, such as text documents or bioinformatic data, that may lack some or all of these strong statistical regularities. In this paper we consider the general question of how to construct deep architectures with small learning complexity on g…

2015-06-16cs.CVcs.LGcs.NE

Leveraging Structured Pruning of Convolutional Neural Networks

Structured pruning is a popular method to reduce the cost of convolutional neural networks, that are the state of the art in many computer vision tasks. However, depending on the architecture, pruning introduces dimensional discrepancies which prevent the actual reduction of pruned networks. To tackle this problem, we propose a method that is able to take any structured pruning mask and generate a network that does not encounter any of these problems and can be leveraged efficiently. We provide an accurate description of our solution and show results of gains, in energy consumption and inferen…

2022-06-13cs.NE

Structured Pruning of Deep Convolutional Neural Networks

Seoul National University（首尔大学）

Real time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also do not fit well on parallel computation. We introduce structured sparsity at various scales for convolutional neural networks, which are channel wise, kernel wise and intra kernel strided sparsity. This structured sparsity is very advantageous for direct computational resource sa…

2015-12-29cs.LGcs.NEstat.ML