SCOUT：用于大语言模型预训练中故障定位的对称共识异常检测
SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training
AI总结：
SCOUT是用于LLM预训练故障定位的统一运行时框架，基于等价副本严格多数共识识别异常，可与主流大模型训练框架集成，解决现有诊断方法的局限。
AI中文摘要：
在大语言模型（LLM）预训练过程中，同步操作会将各进程秩（rank）本地的停顿、速度减慢及数值错误扩散为整个作业的症状，掩盖其来源。现有诊断方法常依赖进程内监控器，无法在训练器阻塞或终止后报告；或依赖仅保留同步症状的事后日志；离线健康测试则会丢失触发故障的工作负载与运行条件。我们提出SCOUT，这是一个统一的运行时故障定位框架，基于核心设计原则：通过等价副本间的严格多数共识识别异常值。该框架对齐副本的进度、时间及数值证据，随后利用其共识集体通信（Consensus Collective Communication, C3）抽象识别出紧凑签名与其他副本不一致的秩。当训练挂起时，带外（out-of-band）CPU观察器仍保持响应；原位重放则利用实时作业的模型状态、内核、内存分配、通信路径及热与内存压力，重现周期性的落后者与静默数据损坏（Silent Data Corruption, SDC）。集体指纹可暴露秩本地的协议分歧。干净的重放覆盖范围可验证检查点的数值完整性，防止恢复时选择被SDC损坏的状态。SCOUT可与PyTorch、TorchTitan、Megatron-Core及DeepSpeed集成，无需修改训练循环或框架源码。SCOUT为开源项目，链接为this https URL。
英文摘要：
In LLM pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin. Existing diagnosis often relies on in-process monitors that cannot report after the trainer blocks or terminates, or on post-mortem logs that preserve only synchronized symptoms; offline health tests lose the workload and operating conditions that triggered the failure. We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas. SCOUT aligns replica progress, timing, and numerical evidence, then uses its Consensus Collective Communication (C3) abstraction to identify ranks whose compact signatures disagree with their peers. An out-of-band CPU observer remains responsive when training hangs, whereas in-situ replay exercises recurring stragglers and silent data corruption (SDC) beside the live job with its model state, kernels, allocations, communication path, and thermal and memory pressure present. Collective fingerprints expose rank-local protocol divergence. Clean replay coverage certifies checkpoint numerical integrity, preventing recovery from selecting state corrupted by SDC. SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications. SCOUT is open source at https://github.com/LMResiliency/lm-resiliency.
