通过数据驱动、预测性、主动式自治实现无操作员数据中心
Towards Operator-less Data Centers Through Data-Driven, Predictive, Proactive Autonomics
摘要：
持续依赖人类操作员来管理数据中心是他们达到极端规模的主要障碍。一般来说，大型计算机系统，特别是数据中心，最终将使用通过数据科学工具获得的预测计算和可执行模型进行管理，在这一点上，人类的干预将仅限于制定高级别的目标和政策，而不是执行低级别的操作。数据驱动的数据中心管理和控制基于使用实时数据构建和更新的整体预测模型，为限制运营商在数据中心中的作用开辟了一条可能的道路。在本文中，我们对在12K节点集群中收集的公共Google数据集进行了数据科学研究，目标是构建和评估节点故障的预测模型。我们的研究结果通过显示基于典型数据中心日志中发现的数据的预测模型的有效性来支持数据驱动方法的实用性。我们使用Google Cloud套件中的大数据SQL平台BigQuery来处理大量数据，并生成一个丰富的特征集，用于描述节点状态随时间的变化。我们描述了一个集成分类器可以建立在许多随机森林分类器，每个训练这些功能，预测如果节点将在未来的24小时窗口失败。我们的评估表明，如果我们限制误报率为5%，我们可以实现27%和88%之间的真阳性率与精度在50%和72%之间变化。这种性能水平允许我们恢复大部分的作业的执行（通过重定向到其他节点时，预测当前节点的故障），否则会被浪费的故障。[...]
英文摘要：
Continued reliance on human operators for managing data centers is a major impediment for them from ever reaching extreme dimensions. Large computer systems in general, and data centers in particular, will ultimately be managed using predictive computational and executable models obtained through data-science tools, and at that point, the intervention of humans will be limited to setting high-level goals and policies rather than performing low-level operations. Data-driven autonomics, where management and control are based on holistic predictive models that are built and updated using live data, opens one possible path towards limiting the role of operators in data centers. In this paper, we present a data-science study of a public Google dataset collected in a 12K-node cluster with the goal of building and evaluating predictive models for node failures. Our results support the practicality of a data-driven approach by showing the effectiveness of predictive models based on data found in typical data center logs. We use BigQuery, the big data SQL platform from the Google Cloud suite, to process massive amounts of data and generate a rich feature set characterizing node state over time. We describe how an ensemble classifier can be built out of many Random Forest classifiers each trained on these features, to predict if nodes will fail in a future 24-hour window. Our evaluation reveals that if we limit false positive rates to 5%, we can achieve true positive rates between 27% and 88% with precision varying between 50% and 72%.This level of performance allows us to recover large fraction of jobs' executions (by redirecting them to other nodes when a failure of the present node is predicted) that would otherwise have been wasted due to failures. [...]
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
