arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.11863cs.LG
收藏

基于反向传播的多层GBDT学习算法

Learning Multi-Layered GBDT Via Back Propagation

School of Electronic Engineering, Xidian University(西安电子科技大学电子工程学院);Xi’an, 710071, China(中国西安(710071))

Zhendong Zhang

摘要:

深度神经网络能够通过反向传播(BP)学习多层表示。虽然梯度提升决策树(GBDT)对于表格数据建模是有效的,但是它相对于其输入是不可微的,从而遭受学习多层表示。在本文中,我们提出了一个框架,学习多层GBDT通过BP。我们基于线性回归来近似GBDT的梯度。具体地说,我们使用线性回归来替换每个叶子上的常数值,忽略单个样本对树结构的贡献。通过这种方式,我们估计了中间表示的梯度,这有助于多层GBDT的BP。实验结果表明,该方法在性能和表示能力方面是有效的。据我们所知,这是第一个工作,优化多层GBDT通过BP。这项工作为探索基于深度树的学习和将GBDT与神经网络相结合提供了新的可能性。

英文摘要:

Deep neural networks are able to learn multi-layered representation via back propagation (BP). Although the gradient boosting decision tree (GBDT) is effective for modeling tabular data, it is non-differentiable with respect to its input, thus suffering from learning multi-layered representation. In this paper, we propose a framework of learning multi-layered GBDT via BP. We approximate the gradient of GBDT based on linear regression. Specifically, we use linear regression to replace the constant value at each leaf ignoring the contribution of individual samples to the tree structure. In this way, we estimate the gradient for intermediate representations, which facilitates BP for multi-layered GBDT. Experiments show the effectiveness of the proposed method in terms of performance and representation ability. To the best of our knowledge, this is the first work of optimizing multi-layered GBDT via BP. This work provides a new possibility of exploring deep tree based learning and combining GBDT with neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑