arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1003.0951cs.DC
收藏

LogMaster:大规模集群系统事件相关性挖掘

LogMaster: Mining Event Correlations in Logs of Large scale Cluster Systems

Rui Ren, Xiaoyu Fu, Jianfeng Zhan, Wei Zhou

摘要:

本文提出了一种方法和一个名为LogMaster的系统,用于挖掘具有多个属性的事件的相关性,即,节点ID、应用ID、事件类型和事件严重性。与传统的交易数据不同,例如,超市购物、系统日志有其独特的特点,因此我们提出了几种创新的方法来挖掘它们之间的相关性。我们提出了一个简单的度量标准来衡量可能发生的事件的相关性。在相关性度量的基础上,提出了两种挖掘事件相关性的方法;同时,提出了一种新的抽象方法:事件相关图(ECG)来表示事件相关性,并给出了一种基于ECG的事件预测算法。针对中国移动研究院基于Hadoop的云计算生产系统和美国洛斯阿拉莫斯国家实验室(LANL)生产HPC集群系统的系统日志,我们在三种场景下评估了我们的方法:(a)基于故障和非故障事件预测所有事件;(b)基于故障和非故障事件仅预测故障事件;(c)在去除非故障事件之后预测故障事件。

英文摘要:

This paper presents a methodology and a system, named LogMaster, for mining correlations of events that have multiple attributions, i.e., node ID, application ID, event type, and event severity, in logs of large-scale cluster systems. Different from traditional transactional data, e.g., supermarket purchases, system logs have their unique characteristic, and hence we propose several innovative approaches to mine their correlations. We present a simple metrics to measure correlations of events that may happen interleavedly. On the basis of the measurement of correlations, we propose two approaches to mine event correlations; meanwhile, we propose an innovative abstraction: event correlation graphs (ECGs) to represent event correlations, and present an ECGs based algorithm for predicting events. For two system logs of a production Hadoop-based cloud computing system at Research Institution of China Mobile and a production HPC cluster system at Los Alamos National Lab (LANL), we evaluate our approaches in three scenarios: (a) predicting all events on the basis of both failure and non-failure events; (b) predicting only failure events on the basis of both failure and non-failure events; (c) predicting failure events after removing non-failure events.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑