arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.00726cs.LG
收藏

探索在LLM训练中作为可靠性挑战的静默数据损坏

Exploring Silent Data Corruption as a Reliability Challenge in LLM Training

Anton Altenbernd, Philipp Wiesner, Odej Kao

AI总结:

研究静默数据损坏对LLM预训练的影响,通过GPU矩阵乘法指令注入故障,分析不同位位置、内核函数和执行阶段的敏感性,并提出轻量检测方法以缓解有害参数更新。

摘要:

随着大型语言模型(LLM)规模和复杂性的增加,训练过程中故障的后果变得越来越严重。静默数据损坏(SDC)是一个主要挑战:由硬件引起的故障绕过系统级检测机制。SDC可能像良性数值噪声一样表现,但也可以导致有害的梯度损坏,导致损失激增、发散或停滞。本文提供了一项受控研究,探讨间歇性SDC对LLM预训练的影响。通过在GPU矩阵乘法指令级别进行有针对性的故障注入,我们分析了不同位位置、内核函数和执行阶段的敏感性。我们的分析表明,本地起源的故障可以产生有影响的损坏,包括NaN传播、损失中的短暂激增、梯度范数和注意力日志的激增,以及持续的参数发散。基于观察到的损坏签名,我们提出了一种轻量级检测方法,以识别可能有害的参数更新。在具有60M、350M和1.3B参数的LLaMA模型上的实验表明,检测到后重新计算最近的训练步骤可以有效缓解这些事件的影响。

英文摘要:

As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑