A feedforward neural network using rectified linear units constructs a mapping from inputs to outputs by partitioning its input space into a set of convex regions where points within a region share a single affine transformation. In order to understand how neural networks work, when and why they fail, and how they compare to biological intelligence, we need to understand the organization and formation of these regions. Step one is to design and implement algorithms for exact region enumeration in networks beyond toy examples.
In this work, we present parallel algorithms for exact enumeration in deep (and shallow) neural networks. Our work has three main contributions: (1) we present a novel algorithm framework and parallel algorithms for region enumeration; (2) we implement one of our algorithms on a variety of network architectures and experimentally show how the number of regions dictates runtime; and (3) we show, using our algorithm's output, how the dimension of a region's affine transformation impacts further partitioning of the region by deeper layers.
To our knowledge, we run our implemented algorithm on networks larger than all of the networks used in the existing region enumeration literature. Further, we experimentally demonstrate the importance of parallelism for region enumeration of any reasonably sized network.
Parallelization Techniques for Verifying Neural Networks
Stanford University（斯坦福大学）；NASA Ames（美国国家航空航天局艾姆斯研究中心）；KBR Inc.（KBR公司）；Carnegie Mellon University（卡内基梅隆大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Discovering physical laws with parallel symbolic enumeration
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；School of Engineering Science, University of Chinese Academy of Sciences（中国科学院大学工程科学学院）；State Key Laboratory of Nonlinear Mechanics, Institute of Mechanics, Chinese Academy of Sciences（中国科学院力学研究所非线性力学重点实验室）；Department of Computer Science and Engineering, HKUST（香港科技大学计算机科学与工程系）
Exploring specialization and sensitivity of convolutional neural networks in the context of simultaneous image augmentations
Marchuk Institute of Numerical Mathematics（马尔库克数值数学研究所）；Skolkovo Institute of Science and Technology（斯克尔科沃科学与技术研究所）；Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University（洛蒙诺索夫莫斯科国立大学计算数学与 cybernetics 学院）；Artificial Intelligence Research Institute (AIRI)（人工智能研究 institute (AIRI)）
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Carnegie Mellon University（卡内基梅隆大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Petuum Inc.（Petuum公司）
Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensive model parallelism. One way to improve training speed is to compress the message size in communication. Previous approaches have primarily focused on compressing gradients in a data parallelism setting, but compression in a model-parallel setting is an understudied area. We have discovered that model parallelism has fundamentally different characteristics than data parallelism. In this work, we present the first emp…
ETH Zürich（苏黎世联邦理工学院）；Stanford University（斯坦福大学）；Zhejiang University（浙江大学）；Carnegie Mellon University（卡内基梅隆大学）；Meta AI
Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive studies of gradient compression for data parallel-style training, compressing the activations for models trained with pipeline parallelism is still an open problem. In this paper, we propose AC-SGD, a novel activation compression algorithm for communication-efficient pipeline parallelism training over slow networks. Different from previous efforts in activation compression, instead of compressing activation values d…
The Click-though Rate (CTR) prediction task is a basic task in recommendation system. Most of the previous researches of CTR models built based on Wide \& deep structure and gradually evolved into parallel structures with different modules. However, the simple accumulation of parallel structures can lead to higher structural complexity and longer training time. Based on the Sigmoid activation function of output layer, the linear addition activation value of parallel structures in the training process is easy to make the samples fall into the weak gradient interval, resulting in the phenomenon …
A 65nm 8b-Activation 8b-Weight SRAM-Based Charge-Domain Computing-in-Memory Macro Using A Fully-Parallel Analog Adder Network and A Single-ADC Interface
Tsinghua University（清华大学）；Institute of Microelectronics of the Chinese Academy of Sciences（中国科学院微电子研究所）；University of Electronic Science and Technology of China（电子科技大学）
Loom: Exploiting Weight and Activation Precisions to Accelerate Convolutional Neural Networks
University of Toronto（多伦多大学）
介绍了一种用于卷积神经网络（CNN）的硬件推理加速器Loom（LM）。在LM中，可以节省的每一位数据精度都转化为成比例的性能增益。具体来说，对于卷积层，LM的执行时间与权重和激活的精度成反比。对于全连接层，LM的性能与权重的精度成反比。LM的目标是面积和带宽受限的片上系统设计，例如那些在移动设备上无法负担存储片上每层所需的多兆字节缓冲区的设计。因此，在给定数据带宽预算的情况下，LM相对于等效的位并行加速器提高了能量效率和性能。对于权重和激活，LM可以利用轮廓导出的每层精度。然而，在运行时，LM进一步以比层粒度小得多的粒度调整激活精度。此外，它可以自然地以比层更小的粒度利用权重精度可变性。平均而言，在几个图像分类CNN中，对于每个周期可以执行相当于128个16 b x 16 b乘法累加操作的配置，LM的性能比最先进的位并行加速器[1]高出4.38倍，而精度没有任何损失，同时能效高出3.54倍。LM可以在执行性能和能源效率的额外改进上权衡准确性，并且与仅针对激活精度的加速器相比是有利的。我们还研究了2位和4位LM变体，发现每周期2位变体是最节能的。
Threshold activation functions are highly preferable in neural networks due to their efficiency in hardware implementations. Moreover, their mode of operation is more interpretable and resembles that of biological neurons. However, traditional gradient based algorithms such as Gradient Descent cannot be used to train the parameters of neural networks with threshold activations since the activation function has zero gradient except at a single non-differentiable point. To this end, we study weight decay regularized training problems of deep neural networks with threshold activations. We first s…
Center of Informatics（信息学中心）；Federal University of Pernambuco（伯南布哥联邦大学）；University of Pernambuco（伯南布哥大学）
The definition of a Neural Network architecture is one of the most critical and challenging tasks to perform. In this paper, we propose ParallelMLPs. ParallelMLPs is a procedure to enable the training of several independent Multilayer Perceptron Neural Networks with a different number of neurons and activation functions in parallel by exploring the principle of locality and parallelization capabilities of modern CPUs and GPUs. The core idea of this technique is to use a Modified Matrix Multiplication that replaces an ordinal matrix multiplication by two simple matrix operations that allow sepa…
