arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1610.01807cs.DCcs.AI
收藏

云系统上的并行大规模属性约简

Parallel Large-Scale Attribute Reduction on Cloud Systems

Southwest Jiaotong University(西南交通大学);Georgia State University(佐治亚州立大学)

Junbo Zhang, Tianrui Li, Yi Pan

摘要:

现代社会新兴信息技术和应用模式的迅速发展,互联网、物联网、云计算和三网融合,已经导致了大数据时代的到来。大数据蕴含着巨大的价值,但由于数据的不确定性和不一致性,从大数据中挖掘知识是一项极具挑战性的任务。属性约简(也称为特征选择)不仅可以作为一个有效的预处理步骤,而且还可以利用数据冗余来减少不确定性。然而,现有的解决方案是1)针对单个机器而设计的,这意味着整个数据必须适合主存储器并且并行性是有限的; 2)或者针对Hadoop平台而设计的,这意味着数据必须频繁地加载到分布式存储器中,因此变得低效。在本文中,我们克服了这些缺点,以最大限度地提高效率,并提出了一个统一的框架,并行大规模属性约简,称为PLAR,大数据分析。PLAR由三个组件组成:1)基于粒度计算(GrC)的初始化:它将决策表(即,原始数据表示)转换为粒度表示,这减少了空间量,因此可以很容易地缓存在分布式存储器中:2)模型并行性:它同时评估所有特征候选,并使属性约简高度并行化; 3)数据并行性:它使用MapReduce风格的方式并行计算属性的重要性。我们在Spark上实现了四种具有代表性的启发式特征选择算法,并在各种大型数据集上进行了评估,包括UCI和天文数据集,发现我们的方法超越现有解决方案的优势。

英文摘要:

The rapid growth of emerging information technologies and application patterns in modern society, e.g., Internet, Internet of Things, Cloud Computing and Tri-network Convergence, has caused the advent of the era of big data. Big data contains huge values, however, mining knowledge from big data is a tremendously challenging task because of data uncertainty and inconsistency. Attribute reduction (also known as feature selection) can not only be used as an effective preprocessing step, but also exploits the data redundancy to reduce the uncertainty. However, existing solutions are designed 1) either for a single machine that means the entire data must fit in the main memory and the parallelism is limited; 2) or for the Hadoop platform which means that the data have to be loaded into the distributed memory frequently and therefore become inefficient. In this paper, we overcome these shortcomings for maximum efficiency possible, and propose a unified framework for Parallel Large-scale Attribute Reduction, termed PLAR, for big data analysis. PLAR consists of three components: 1) Granular Computing (GrC)-based initialization: it converts a decision table (i.e., original data representation) into a granularity representation which reduces the amount of space and hence can be easily cached in the distributed memory: 2) model-parallelism: it simultaneously evaluates all feature candidates and makes attribute reduction highly parallelizable; 3) data-parallelism: it computes the significance of an attribute in parallel using a MapReduce-style manner. We implement PLAR with four representative heuristic feature selection algorithms on Spark, and evaluate them on various huge datasets, including UCI and astronomical datasets, finding our method's advantages beyond existing solutions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑