探索 Winograd 卷积以实现高性价比的神经网络容错
Exploring Winograd Convolution for Cost-effective Neural Network Fault Tolerance
- Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
- University of Chinese Academy of Sciences（中国科学院大学）
- Beijing Institute of Control Engineering（北京控制工程研究所）
- Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文首次从多粒度评估 Winograd 卷积容错性，并将其与 TMR、故障感知重训练及受限激活函数结合，以更低开销提升神经网络抗软错误能力。
AI中文摘要：
Winograd 通常因能减少乘法运算而被用于优化卷积性能和计算效率，但其带来的可靠性问题往往被忽视。在本工作中，我们观察到 Winograd 卷积在提升神经网络（NN）容错能力方面的巨大潜力。基于这一观察，我们首次从模型、层和操作类型等不同粒度全面评估了 Winograd 卷积的容错性。随后，我们探索利用 Winograd 卷积的固有容错能力，以高性价比的方式保护神经网络免受软错误影响。具体而言，我们主要研究如何将 Winograd 卷积与经典容错设计方法有效结合，包括三模冗余（TMR）、故障感知重训练和受限激活函数。实验表明，与标准卷积相比，Winograd 卷积可在不造成任何精度损失的情况下，将容错设计开销平均降低 55.77%；若进一步考虑 Winograd 卷积的固有容错能力，还可将计算开销降低 17.24%。当将其应用于通过故障感知重训练和受限激活函数增强的容错神经网络时，所得模型在各类故障存在时的精度通常表现出显著提升。
英文摘要：
Winograd is generally utilized to optimize convolution performance and computational efficiency because of the reduced multiplication operations, but the reliability issues brought by winograd are usually overlooked. In this work, we observe the great potential of winograd convolution in improving neural network (NN) fault tolerance. Based on the observation, we evaluate winograd convolution fault tolerance comprehensively from different granularities ranging from models, layers, and operation types for the first time. Then, we explore the use of inherent fault tolerance of winograd convolution for cost-effective NN protection against soft errors. Specifically, we mainly investigate how winograd convolution can be effectively incorporated with classical fault-tolerant design approaches including triple modular redundancy (TMR), fault-aware retraining, and constrained activation functions. According to our experiments, winograd convolution can reduce the fault-tolerant design overhead by 55.77\% on average without any accuracy loss compared to standard convolution, and further reduce the computing overhead by 17.24\% when the inherent fault tolerance of winograd convolution is considered. When it is applied on fault-tolerant neural networks enhanced with fault-aware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in presence of various faults.
