Deep neural networks (DNNs) are state-of-the-art algorithms for multiple applications, spanning from image classification to speech recognition. While providing excellent accuracy, they often have enormous compute and memory requirements. As a result of this, quantized neural networks (QNNs) are increasingly being adopted and deployed especially on embedded devices, thanks to their high accuracy, but also since they have significantly lower compute and memory requirements compared to their floating point equivalents. QNN deployment is also being evaluated for safety-critical applications, such as automotive, avionics, medical or industrial. These systems require functional safety, guaranteeing failure-free behaviour even in the presence of hardware faults. In general fault tolerance can be achieved by adding redundancy to the system, which further exacerbates the overall computational demands and makes it difficult to meet the power and performance requirements. In order to decrease the hardware cost for achieving functional safety, it is vital to explore domain-specific solutions which can exploit the inherent features of DNNs. In this work we present a novel methodology called fault-aware training (FAT), which includes error modeling during neural network (NN) training, to make QNNs resilient to specific fault models on the device. Our experiments show that by injecting faults in the convolutional layers during training, highly accurate convolutional neural networks (CNNs) can be trained which exhibits much better error tolerance compared to the original. Furthermore, we show that redundant systems which are built from QNNs trained with FAT achieve higher worse-case accuracy at lower hardware cost. This has been validated for numerous classification tasks including CIFAR10, GTSRB, SVHN and ImageNet.
Technische Universität Dresden（德累斯顿工业大学）；University of Stuttgart（斯图加特大学）；Robert Bosch GmbH（罗伯特·博世有限公司）；Bosch (China) Investment Ltd.（博世（中国）投资有限公司）
Today, Deep Learning (DL) enhances almost every industrial sector, including safety-critical areas. The next generation of safety standards will define appropriate verification techniques for DL-based applications and propose adequate fault tolerance mechanisms. DL-based applications, like any other software, are susceptible to common random hardware faults such as bit flips, which occur in RAM and CPU registers. Such faults can lead to silent data corruption. Therefore, it is crucial to develop methods and tools that help to evaluate how DL components operate under the presence of such faults…
The functionality of electronic circuits can be seriously impaired by the occurrence of dynamic hardware faults. Particularly, for digital ultra low-power systems, a reduced safety margin can increase the probability of dynamic failures. This work investigates a deep learning approach to mitigate dynamic fault impact for artificial neural networks. As a theoretic use case, image compression by means of a deep autoencoder is considered. The evaluation shows a linear dependency of the test loss to the fault injection rate during testing. If the number of training epochs is sufficiently large, ou…
Intel（英特尔）；University of British Columbia（不列颠哥伦比亚大学）
Convolutional neural networks (CNNs) have become an established part of numerous safety-critical computer vision applications, including human robot interactions and automated driving. Real-world implementations will need to guarantee their robustness against hardware soft errors corrupting the underlying platform memory. Based on the previously observed efficacy of activation clipping techniques, we build a prototypical safety case for classifier CNNs by demonstrating that range supervision represents a highly reliable fault detector and mitigator with respect to relevant bit flips, adopting …
University of Maryland, College Park（马里兰大学帕克分校）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）
Deep neural networks (DNNs) have been shown to tolerate "brain damage": cumulative changes to the network's parameters (e.g., pruning, numerical perturbations) typically result in a graceful degradation of classification accuracy. However, the limits of this natural resilience are not well understood in the presence of small adversarial changes to the DNN parameters' underlying memory representation, such as bit-flips that may be induced by hardware fault attacks. We study the effects of bitwise corruptions on 19 DNN models---six architectures on three image classification tasks---and we show …
Technische Universität Wien (TU Wien)（维也纳工业大学）；New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校）
Spiking neural networks (SNNs) have shown a potential for having low energy with unsupervised learning capabilities due to their biologically-inspired computation. However, they may suffer from accuracy degradation if their processing is performed under the presence of hardware-induced faults in memories, which can come from manufacturing defects or voltage-induced approximation errors. Since recent works still focus on the fault-modeling and random fault injection in SNNs, the impact of memory faults in SNN hardware architectures on accuracy and the respective fault-mitigation techniques are …
Thapar Institute of Engineering and Technology（塔帕尔工程技术学院）；PES College of Engineering（PES工程学院）
Today, Neural Networks are the basis of breakthroughs in virtually every technical domain. Their application to accelerators has recently resulted in better performance and efficiency in these systems. At the same time, the increasing hardware failures due to the latest (shrinked) semiconductor technology needs to be addressed. Since accelerator systems are often used to back time-critical applications such as self-driving cars or medical diagnosis applications, these hardware failures must be eliminated. Our research evaluates these failures from a systemic point of view. Based on our results…
Enhancing Hardware Fault Tolerance in Machines with Reinforcement Learning Policy Gradient Algorithms
Department of Computing Science, University of Alberta（阿尔伯塔大学计算机科学系）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）；Advanced Technology R&D Center, Mitsubishi Electric Corporation（三菱电机株式会社先进技术研发中心）；Information Technology R&D Center, Mitsubishi Electric Corporation（三菱电机株式会社信息技术研发中心）
University of Missouri（密苏里大学）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
Systolic array-based deep neural network (DNN) accelerators have recently gained prominence for their low computational cost. However, their high energy consumption poses a bottleneck to their deployment in energy-constrained devices. To address this problem, approximate computing can be employed at the cost of some tolerable accuracy loss. However, such small accuracy variations may increase the sensitivity of DNNs towards undesired subtle disturbances, such as permanent faults. The impact of permanent faults in accurate DNNs has been thoroughly investigated in the literature. Conversely, the…
Exploration of Activation Fault Reliability in Quantized Systolic Array-Based DNN Accelerators
Tallinn University of Technology（塔林理工大学）；University of Alberta（阿尔伯塔大学）；Shahid Bahonar University of Kerman（克尔曼沙希德·巴霍纳尔大学）；York University（约克大学）；Mälardalen University（梅拉达伦大学）
Technische Universität Wien (TU Wien)（维也纳工业大学（TU Wien））；New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校（NYUAD））
Specialized hardware accelerators have been designed and employed to maximize the performance efficiency of Spiking Neural Networks (SNNs). However, such accelerators are vulnerable to transient faults (i.e., soft errors), which occur due to high-energy particle strikes, and manifest as bit flips at the hardware layer. These errors can change the weight values and neuron operations in the compute engine of SNN accelerators, thereby leading to incorrect outputs and accuracy degradation. However, the impact of soft errors in the compute engine and the respective mitigation techniques have not be…
Technische Universität Wien (TU Wien)（维也纳工业大学（TU Wien））
Deep Neural Networks (DNNs) are widely being adopted for safety-critical applications, e.g., healthcare and autonomous driving. Inherently, they are considered to be highly error-tolerant. However, recent studies have shown that hardware faults that impact the parameters of a DNN (e.g., weights) can have drastic impacts on its classification accuracy. In this paper, we perform a comprehensive error resilience analysis of DNNs subjected to hardware faults (e.g., permanent faults) in the weight memory. The outcome of this analysis is leveraged to propose a novel error mitigation technique which …
Special Session: Approximation and Fault Resiliency of DNN Accelerators
Tallinn University of Technology（塔林理工大学）；University of Naples Federico II（那不勒斯费德里科二世大学）；Ecole Centrale de Lyon（里昂中央理工学院）；Mälardalen University（马拉达伦大学）；Politecnico di Torino（都灵理工大学）
MRFI: An Open Source Multi-Resolution Fault Injection Framework for Neural Network Processing
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Control Engineering（北京控制工程研究所）
Georgia Institute of Technology（佐治亚理工学院）；Twentieth Century Fox（二十世纪福克斯公司）；Starfleet Academy（星际舰队学院）；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；Tyrell Inc.（泰瑞尔公司）
Machine Learning (ML) is making a strong resurgence in tune with the massive generation of unstructured data which in turn requires massive computational resources. Due to the inherently compute- and power-intensive structure of Neural Networks (NNs), hardware accelerators emerge as a promising solution. However, with technology node scaling below 10nm, hardware accelerators become more susceptible to faults, which in turn can impact the NN accuracy. In this paper, we study the resilience aspects of Register-Transfer Level (RTL) model of NN accelerators, in particular, fault characterization a…
Northeastern University（东北大学）；North Dakota State University（北达科他州立大学）；Washington State University（华盛顿州立大学）；University of Connecticut（康涅狄格大学）；Chongqing University（重庆大学）；University of South Alabama（南阿拉巴马大学）
Recent research demonstrated the promise of using resistive random access memory (ReRAM) as an emerging technology to perform inherently parallel analog domain in-situ matrix-vector multiplication -- the intensive and key computation in deep neural networks (DNNs). However, hardware failure, such as stuck-at-fault defects, is one of the main concerns that impedes the ReRAM devices to be a feasible solution for real implementations. The existing solutions to address this issue usually require an optimization to be conducted for each individual device, which is impractical for mass-produced prod…
