NVIDIA MPS 中基于特征分析的 GPU 容错能力
Characterization-Guided GPU Fault Resilience in NVIDIA MPS
AI总结:
针对 NVIDIA MPS 容错性弱的问题,通过系统分析 GPU 故障特征,设计了故障隔离和基于虚拟内存的快速恢复两种机制,有效处理故障且开销小。
摘要:
NVIDIA 多进程服务 (MPS) 通过允许多个进程在同一 GPU 上并发执行,实现了细粒度的 GPU 共享,是提高 GPU 利用率的重要机制。然而,MPS 的容错能力较弱:一个进程中的故障可能导致所有协同运行的进程终止,这限制了其在多租户 GPU 集群等对容错性要求较高的场景中的应用。本文设计了容错 MPS 来解决这一问题。我们的设计基于对 GPU 故障的系统特征分析及其端到端处理流程的深入分析。基于这些洞察,我们设计了两种互补机制。一种故障隔离机制,用于处理占主导地位的与内存相关的故障,这些故障可以通过开放 GPU 驱动内核模块中的软件干预完全隔离。对于其他故障,其处理过程位于专有软件中,我们设计了一种实用机制——基于虚拟内存的 GPU 驻留状态共享的快速恢复。我们在不同 GPU 和工作负载上的评估表明,这些机制能够有效处理相应故障,且开销极小。
英文摘要:
NVIDIA Multi-Process Service (MPS) enables fine-grained GPU sharing by allowing multiple processes to execute concurrently on the same GPU, making it an important mechanism for improving GPU utilization. However, MPS has weak fault resilience: a fault in one process can terminate all co-running processes, limiting its adoption in resilience-critical settings such as multi-tenant GPU clusters. In this work, we design fault-resilient MPS to solve this problem. Our design is guided by insights from a systematic characterization of GPU faults and a deep analysis of their end-to-end processing pipeline. Based on these insights, we design two complementary mechanisms. First, we design a fault isolation mechanism for the dominant memory-related faults that can be fully isolated while preserving process-level fail-stop semantics by software intervention in the open GPU driver kernel module. For other faults whose process is within proprietary software, we design a fast-recovery substrate that combines virtual-memory-based GPU-resident state sharing with pre-initialized standbys. Our evaluation across GPUs and workloads demonstrates effective fault isolation and fast recovery with minimal overhead: in an end-to-end case study, isolation incurs no visible outage, while recovery restores pre-fault throughput in 355\,ms.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。