GPUAlert: 一种零侵入的进程边界监控器，用于诊断GPU训练任务故障
GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures
AI总结：
提出GPUAlert，一种零侵入的进程边界监控工具，通过预启动日志保证、通知器隔离和非静默工件预算三个可靠性原语，在无需修改训练脚本的情况下，发送结构化故障通知，并附带分类原因、持久日志和输出工件。
摘要：
GPU训练任务经常失败，在大型生产集群中大约五分之二的任务会失败，但操作员通常只能在数小时后重新连接时才得知故障。实验跟踪器需要编辑训练脚本并保持云连接；调度器的邮件钩子只传递一行状态信息，没有原因和日志。GPUAlert是一个命令行包装器，它在进程边界监控任何训练命令，无需修改该命令，在完成时发送结构化通知，包含分类的故障原因、持久日志和输出工件。该工具围绕三个可靠性原语组织：预启动日志保证，在子进程崩溃前建立持久目标；通知器隔离，使包装器的退出代码成为子进程状态的纯函数，无论邮件是否成功；以及非静默工件预算，限制附件大小而不会静默丢弃输出。我们发布了一个包含474个GPU训练日志的标注语料库，涵盖15个故障类别，以及一个可复现的评估框架。在十二个硬件可复现的类别上，有序规则分类器达到了0.997的宏F1分数，而无序关键词匹配为0.830，退出代码检查为0.133。包装器开销恒定约为每个任务3毫秒；预启动保证保留了日志，而shell重定向则无输出；在所有15种故障模式下，即使SMTP中继不可达，包装器也返回子进程的原始退出代码。
英文摘要：
GPU training jobs fail often, roughly two in five on large production clusters, yet the operator typically learns of a failure only by reconnecting hours later. Experiment trackers require editing the training script and maintaining a cloud connection; the scheduler's mail hook delivers a single status line with no cause and no logs. GPUAlert is a command-line wrapper that monitors any training command at the process boundary, and with no change to that command, emails a structured notification on completion carrying a classified failure cause, durable logs, and output artifacts. The tool is organized around three reliability primitives: a pre-launch log guarantee that establishes the durable destination before the child process can crash, notifier isolation that makes the wrapper's exit code a pure function of the child's status regardless of whether the email succeeds, and a non-silent artifact budget that bounds attachment size without ever dropping output silently. We release a labelled corpus of 474 GPU training logs across 15 failure classes and a reproducible evaluation harness. On the twelve hardware-reproduced classes, the ordered-rule classifier reaches 0.997 macro-F1, against 0.830 for unordered keyword matching and 0.133 for exit-code inspection. Wrapper overhead is a constant approximately 3ms per job; the pre-launch guarantee preserves a log where a shell redirect yields nothing; and across all 15 failure modes the wrapper returns the child's exit code unchanged even when the SMTP relay is unreachable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
