arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1611.01276cs.LG
收藏

一种通信高效的决策树并行算法

A Communication-Efficient Parallel Algorithm for Decision Tree

Peking University(北京大学);Microsoft Research(微软研究院);Chinese Academy of Mathematics and Systems Science(中国科学院数学与系统科学研究院)

Qi Meng, Guolin Ke, Taifeng Wang, Wei Chen, Qiwei Ye, Zhi-Ming Ma, Tie-Yan Liu

摘要:

决策树(及其扩展,如梯度提升决策树和随机森林)是一种广泛使用的机器学习算法,由于其实际有效性和模型可解释性。随着大数据的出现,越来越需要并行化决策树的训练过程。然而,沿着这条线的大多数现有尝试遭受高通信成本。在本文中,我们提出了一种新的算法,称为\emph{Parallel Voting Decision Tree (PV-Tree)},以解决这一挑战。在将训练数据划分成多个(例如,$M$)机器,该算法在每次迭代中执行局部投票和全局投票。对于本地投票,根据每台计算机的本地数据从每台计算机中选择顶级$k$属性。然后,通过在这些局部候选中的多数投票来确定全局最佳$2k$属性。最后,从本地机器收集全局顶级$2k$属性的全粒度直方图,以识别最佳(信息量最大)属性及其分割点。PV-Tree可以实现非常低的通信成本(与属性总数无关),因此可以很好地扩展。理论分析表明,该算法能够以较大的概率找到最优的属性,从而能够学习出接近最优的决策树。我们在真实数据集上的实验表明,PV-Tree在准确性和效率之间的权衡上明显优于现有的并行决策树算法。

英文摘要:

Decision tree (and its extensions such as Gradient Boosting Decision Trees and Random Forest) is a widely used machine learning algorithm, due to its practical effectiveness and model interpretability. With the emergence of big data, there is an increasing need to parallelize the training process of decision tree. However, most existing attempts along this line suffer from high communication costs. In this paper, we propose a new algorithm, called \emph{Parallel Voting Decision Tree (PV-Tree)}, to tackle this challenge. After partitioning the training data onto a number of (e.g., $M$) machines, this algorithm performs both local voting and global voting in each iteration. For local voting, the top-$k$ attributes are selected from each machine according to its local data. Then, globally top-$2k$ attributes are determined by a majority voting among these local candidates. Finally, the full-grained histograms of the globally top-$2k$ attributes are collected from local machines in order to identify the best (most informative) attribute and its split point. PV-Tree can achieve a very low communication cost (independent of the total number of attributes) and thus can scale out very well. Furthermore, theoretical analysis shows that this algorithm can learn a near optimal decision tree, since it can find the best attribute with a large probability. Our experiments on real-world datasets show that PV-Tree significantly outperforms the existing parallel decision tree algorithms in the trade-off between accuracy and efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑