arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1807.02258cs.AIcs.DB
收藏

基于Spark的大数据集可扩展形式概念分析算法

Scalable Formal Concept Analysis algorithm for large datasets using Spark

School of Information Technology & Engineering, Vellore Institute of Technology(韦洛尔理工学院信息技术与工程学院)

Raghavendra K Chunduri, Aswani Kumar Cherukuri

摘要:

在利用形式概念分析进行大数据集的知识发现和表示过程中,复杂性在识别所有形式概念和构造概念格(概念的有向图)方面起着重要作用。为了在非常大的数据集中识别形式概念并从识别的概念构造有向图,文献中有各种分布式算法。然而,现有的分布式算法不太适合概念生成,因为它是一个迭代的过程。现有的算法都是使用MapReduce和Open MP等分布式框架实现的,这些框架不适合迭代应用。因此,在本文中,我们提出了有效的分布式算法,在大型正式上下文中使用Apache Spark的正式概念生成和概念格有向图的建设。对所提出的算法进行了性能评价,结果表明,与现有算法相比,所提出的算法在概念生成和格图构造方面是有效的.

英文摘要:

In the process of knowledge discovery and representation in large datasets using formal concept analysis, complexity plays a major role in identifying all the formal concepts and constructing the concept lattice(digraph of the concepts). For identifying the formal concepts and constructing the digraph from the identified concepts in very large datasets, various distributed algorithms are available in the literature. However, the existing distributed algorithms are not very well suitable for concept generation because it is an iterative process. The existing algorithms are implemented using distributed frameworks like MapReduce and Open MP, these frameworks are not appropriate for iterative applications. Hence, in this paper we proposed efficient distributed algorithms for both formal concept generation and concept lattice digraph construction in large formal contexts using Apache Spark. Various performance metrics are considered for the evaluation of the proposed work, the results of the evaluation proves that the proposed algorithms are efficient for concept generation and lattice graph construction in comparison with the existing algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑