大规模的无声数据损坏
Silent Data Corruptions at Scale
摘要:
静默数据损坏(SDC)可能会对大规模基础设施服务产生负面影响。SDC不被中央处理单元(CPU)内的错误报告机制捕获,因此在硬件级别上不可追踪。但是,数据损坏会跨堆栈传播,并表现为应用程序级问题。这些类型的错误可能导致数据丢失,并可能需要数月的调试工程时间。在本文中,我们描述了常见的缺陷类型观察到的硅制造,导致SDC。我们讨论一个数据中心应用程序中静默数据损坏的真实示例。我们提供的调试流程遵循的根本原因和分流错误的指令在CPU使用的案例研究,作为一个例子,如何调试这类错误。我们提供了一个高层次的缓解措施概述,以降低大型生产车队内的无声数据损坏的风险。在我们的大规模基础设施中,我们已经在我们的车队中的数十万台机器上运行了一个巨大的无声错误测试场景库。这导致数百个CPU检测到这些错误,表明SDC是跨代的系统性问题。我们对SDC的监测时间超过18个月。基于这一经验,我们确定减少静默数据损坏不仅需要硬件弹性和生产检测机制,还需要强大的容错软件架构。
英文摘要:
Silent Data Corruption (SDC) can have negative impact on large-scale infrastructure services. SDCs are not captured by error reporting mechanisms within a Central Processing Unit (CPU) and hence are not traceable at the hardware level. However, the data corruptions propagate across the stack and manifest as application-level problems. These types of errors can result in data loss and can require months of debug engineering time. In this paper, we describe common defect types observed in silicon manufacturing that leads to SDCs. We discuss a real-world example of silent data corruption within a datacenter application. We provide the debug flow followed to root-cause and triage faulty instructions within a CPU using a case study, as an illustration on how to debug this class of errors. We provide a high-level overview of the mitigations to reduce the risk of silent data corruptions within a large production fleet. In our large-scale infrastructure, we have run a vast library of silent error test scenarios across hundreds of thousands of machines in our fleet. This has resulted in hundreds of CPUs detected for these errors, showing that SDCs are a systemic issue across generations. We have monitored SDCs for a period longer than 18 months. Based on this experience, we determine that reducing silent data corruptions requires not only hardware resiliency and production detection mechanisms, but also robust fault-tolerant software architectures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。