通过Cielo中的故障注入实验了解故障场景和影响
Understanding Fault Scenarios and Impacts through Fault Injection Experiments in Cielo
摘要:
我们提出了一套故障注入实验上ACES(LANL/SNL)Cray XE超级计算机Cielo。我们使用这个实验活动,以提高故障原因和传播,我们观察到在现场故障数据分析NCSA的蓝水的理解。我们使用从日志和网络性能计数器数据中收集的数据1)描述Gemini网络和Cray计算节点中的故障-错误-故障序列和恢复机制,2)了解故障对不同规模的系统和用户应用程序的影响,以及3)识别和重新创建导致不可恢复故障的故障场景,以便为系统和应用程序设计创建新的测试。故障通过特殊的输入命令注入,以中断网络链路、定向连接、节点和刀片。我们提出的扩展,将需要应用我们的方法注入和分析的克雷XC(白羊座)系统。
英文摘要:
We present a set of fault injection experiments performed on the ACES (LANL/SNL) Cray XE supercomputer Cielo. We use this experimental campaign to improve the understanding of failure causes and propagation that we observed in the field failure data analysis of NCSA's Blue Waters. We use the data collected from the logs and from network performance counter data 1) to characterize the fault-error-failure sequence and recovery mechanisms in the Gemini network and in the Cray compute nodes, 2) to understand the impact of failures on the system and the user applications at different scale, and 3) to identify and recreate fault scenarios that induce unrecoverable failures, in order to create new tests for system and application design. The faults were injected through special input commands to bring down network links, directional connections, nodes, and blades. We present extensions that will be needed to apply our methodologies of injection and analysis to the Cray XC (Aries) systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。