V-ABFT：基于方差的自适应阈值用于混合精度深度学习中的容错矩阵乘法
V-ABFT: Variance-Based Adaptive Threshold for Fault-Tolerant Matrix Multiplication in Mixed-Precision Deep Learning
AI总结：
V-ABFT通过基于方差的自适应阈值算法，提升了混合精度深度学习中矩阵乘法的容错性能，实现更精确的误差检测和更低的阈值与误差比。
摘要：
算法基于容错（ABFT）被广泛采用，用于检测深度学习系统中矩阵乘法中的静默数据损坏（SDCs）。然而，现有的阈值确定方法面临关键挑战：分析性边界过于保守，而概率方法如A-ABFT产生的阈值比实际舍入误差大160到4200倍。我们提出了V-ABFT，一种基于方差的自适应阈值算法，通过直接建模验证差值来实现更紧的误差边界。通过利用统计方差估计，V-ABFT将阈值与实际误差比降低到FP32/FP64约为7-20倍，BF16为48-158倍，相较于A-ABFT有6-48倍的改进，同时在BF16、FP16、FP32和FP64精度下保持零误报率。此外，我们证明对于在输出量化前验证的融合内核ABFT实现，低精度GEMM可以使用FP32级别的阈值（e_max≈10^-6），相较于离线验证低精度输出（e_max≈10^-3）实现约1000倍的更精细的检测粒度。我们复现了A-ABFT的实验设置，并验证了我们的实现与原始论文结果的一致性。我们的方法仅需O(n)的复杂度，使用最大/最小/平均统计，相较于A-ABFT的O(pn)寻找p个最大值的复杂度。在合成数据和真实模型权重（LLaMA-7B、GPT-2、ViT）上的广泛实验展示了V-ABFT在各种分布中的有效性。V-ABFT是平台无关的，并已集成到NPUs和GPUs上的容错GEMM实现中。
英文摘要：
Algorithm-Based Fault Tolerance (ABFT) is widely adopted to detect silent data corruptions (SDCs) in matrix multiplication, a cornerstone operation in deep learning systems. However, existing threshold determination methods face critical challenges: analytical bounds are overly conservative, while probabilistic approaches like A-ABFT yield thresholds $160$--$4200\times$ larger than actual rounding errors. We present V-ABFT, a variance-based adaptive threshold algorithm that achieves tighter error bounds by directly modeling the verification difference. By leveraging statistical variance estimation, V-ABFT reduces the threshold-to-actual-error ratio to approximately $7$--$20\times$ for FP32/FP64 and $48$--$158\times$ for BF16, representing a \textbf{6--48$\times$ improvement} over A-ABFT while maintaining zero false positive rate across BF16, FP16, FP32, and FP64 precisions. Furthermore, we demonstrate that for fused-kernel ABFT implementations that verify before output quantization, low-precision GEMM can use FP32-level thresholds ($e_{\max} \approx 10^{-6}$), enabling \textbf{$\sim$1000$\times$ finer detection granularity} compared to offline verification with low-precision output ($e_{\max} \approx 10^{-3}$). We reproduce A-ABFT's experimental setup and validate our implementation against the original paper's results. Our method requires only $O(n)$ complexity using max/min/mean statistics, compared to A-ABFT's $O(pn)$ for finding $p$ largest values. Extensive experiments on synthetic data and real model weights (LLaMA-7B, GPT-2, ViT) demonstrate V-ABFT's effectiveness across diverse distributions. V-ABFT is platform-agnostic and has been integrated into fault-tolerant GEMM implementations on both NPUs and GPUs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
