ProAct：渐进训练用于混合截断激活函数以增强DNNs的鲁棒性
ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs
AI总结：
ProAct通过混合神经元级和层级截断方法，提升DNNs在高比特误差率下的鲁棒性，采用渐进训练优化阈值。
摘要：
深度神经网络（DNNs）被广泛应用于安全性至关重要的应用中，确保硬件可靠性是主要关注点。为了提高DNNs在硬件故障中的可靠性，激活限制技术显著减轻了DNN结构层面的故障影响，无论加速器架构如何。最先进的方法提供要么神经元级要么层级的截断激活函数。它们尝试通过启发式和学习方法确定最优截断阈值。层级截断激活函数在高比特误差率下无法保持DNNs的鲁棒性。另一方面，神经元级截断激活函数由于添加参数而引入了相当大的内存开销，这会增加其对故障的易感性。此外，基于启发式的优化方法需要在搜索过程中进行大量故障注入，导致阈值识别过程耗时。另一方面，同时训练整个层阈值的学习方法往往会产生次优结果。在本工作中，首先，我们证明在DNNs中在整个层中使用神经元级激活函数并非必要。然后，我们提出了一种混合截断激活函数，该函数整合了神经元级和层级方法，仅在DNNs的最后一层应用神经元级截断。此外，为了在截断激活函数中获得最佳阈值，我们引入了ProAct，一种渐进训练方法。这种方法在层级基础上迭代训练阈值，旨在在每层中分别获得最佳阈值。
英文摘要：
Deep Neural Networks (DNNs) are extensively employed in safety-critical applications where ensuring hardware reliability is a primary concern. To enhance the reliability of DNNs against hardware faults, activation restriction techniques significantly mitigate the fault effects at the DNN structure level, irrespective of accelerator architectures. State-of-the-art methods offer either neuron-wise or layer-wise clipping activation functions. They attempt to determine optimal clipping thresholds using heuristic and learning-based approaches. Layer-wise clipped activation functions cannot preserve DNNs resilience at high bit error rates. On the other hand, neuron-wise clipping activation functions introduce considerable memory overhead due to the addition of parameters, which increases their vulnerability to faults. Moreover, the heuristic-based optimization approach demands numerous fault injections during the search process, resulting in time-consuming threshold identification. On the other hand, learning-based techniques that train thresholds for entire layers concurrently often yield sub-optimal results. In this work, first, we demonstrate that it is not essential to incorporate neuron-wise activation functions throughout all layers in DNNs. Then, we propose a hybrid clipped activation function that integrates neuron-wise and layer-wise methods that apply neuron-wise clipping only in the last layer of DNNs. Additionally, to attain optimal thresholds in the clipping activation function, we introduce ProAct, a progressive training methodology. This approach iteratively trains the thresholds on a layer-by-layer basis, aiming to obtain optimal threshold values in each layer separately.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
