arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1802.05421cs.DC
收藏

CADDeLaG:大型密集图序列中的分布式异常检测框架

CADDeLaG: Framework for distributed anomaly detection in large dense graph sequences

Aniruddha Basak, Kamalika Das, Ole J. Mengshoel

摘要:

图的基于随机游走的距离度量,例如通勤时间距离,在各种图算法中是有用的,例如聚类、异常检测和创建低维嵌入。由于这种措施取决于图的谱分解,计算成为一个瓶颈,为大型图,不容易扩展到图,不能加载到内存中。大多数现有的大型图的图挖掘库要么采取采样或利用这种图的稀疏结构进行谱分析。然而,这样的方法不工作的稠密图构建的研究数据集中的实体之间的成对关系。这种研究的例子包括分析网格气候数据中的成对位置,以发现长距离气候现象。这些图形表示通过构造完全连接,并且不能在不丢失有意义信息的情况下稀疏化。在本文中,我们描述了CADDeLaG,一个框架的可扩展计算的通勤时间的距离为基础的异常检测在大型密集的图形,而不需要在内存中加载整个图形。该框架依赖于Apache Spark的以内存为中心的集群计算基础设施,由两个构建块组成:用于通勤时间距离计算的可分解算法和分布式线性系统求解器。我们说明了CADDeLaG的可扩展性和依赖于各种因素,使用合成和真实世界的数据集。我们证明了CADDeLaG在识别气候图序列中的异常方面的有用性,这些异常在历史上由于临时图稀疏化和选举捐赠数据集而被错过。

英文摘要:

Random walk based distance measures for graphs such as commute-time distance are useful in a variety of graph algorithms, such as clustering, anomaly detection, and creating low dimensional embeddings. Since such measures hinge on the spectral decomposition of the graph, the computation becomes a bottleneck for large graphs and do not scale easily to graphs that cannot be loaded in memory. Most existing graph mining libraries for large graphs either resort to sampling or exploit the sparsity structure of such graphs for spectral analysis. However, such methods do not work for dense graphs constructed for studying pairwise relationships among entities in a data set. Examples of such studies include analyzing pairwise locations in gridded climate data for discovering long distance climate phenomena. These graphs representations are fully connected by construction and cannot be sparsified without loss of meaningful information. In this paper we describe CADDeLaG, a framework for scalable computation of commute-time distance based anomaly detection in large dense graphs without the need to load the entire graph in memory. The framework relies on Apache Spark's memory-centric cluster-computing infrastructure and consists of two building blocks: a decomposable algorithm for commute time distance computation and a distributed linear system solver. We illustrate the scalability of CADDeLaG and its dependency on various factors using both synthetic and real world data sets. We demonstrate the usefulness of CADDeLaG in identifying anomalies in a climate graph sequence, that have been historically missed due to ad hoc graph sparsification and on an election donation data set.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑