arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.05942cs.LGmath.OC
收藏

软回归树:一种模型变体与一种分解训练算法

Soft regression trees: a model variant and a decomposition training algorithm

Politecnico di Milano(米兰理工大学)

Antonio Consolo, Edoardo Amaldi, Andrea Manno

AI总结:

提出一种软多变量回归树变体及其分解训练算法,通过条件计算与聚类初始化实现高效训练,在15个数据集上展现出更高准确性、鲁棒性及显著降低的训练时间。

摘要:

决策树因其可解释性和良好的准确性,被广泛应用于各种应用领域的分类和回归任务中。在过去十年中,越来越多的关注投向了在分支节点具有确定性或软分裂规则的全局优化决策树,这些树通过在所有树参数上优化误差函数来进行训练。在这项工作中,我们提出了一种新的软多变量回归树(SRTs)变体,其中对于每个输入向量,预测被定义为与单个叶子节点相关联的线性回归,即通过将输入向量从根节点沿着概率较高的分支进行路由所获得的叶子节点。SRTs表现出条件计算特性,即每次预测仅依赖于少量节点(参数),并且我们用于训练它们的非线性优化公式易于分解。在展示了SRTs的通用近似结果之后,我们提出了一种分解训练算法,包括基于聚类的初始化过程和一种沿树重新分配输入向量的启发式方法。在温和的假设下,我们建立了渐近收敛保证。在15个知名数据集上的实验表明,与使用Blanquero等人的非线性优化公式训练的传统软回归树相比,我们的SRTs和分解算法产生了更高的准确性和鲁棒性;与Bertsimas和Dunn的混合整数优化方法相比,训练时间显著减少,平均准确性也略好。我们还报告了与Random Forest集成方法的比较。

英文摘要:

Decision trees are widely used for classification and regression tasks in a variety of application fields due to their interpretability and good accuracy. During the past decade, growing attention has been devoted to globally optimized decision trees with deterministic or soft splitting rules at branch nodes, which are trained by optimizing the error function over all the tree parameters. In this work, we propose a new variant of soft multivariate regression trees (SRTs) where, for every input vector, the prediction is defined as the linear regression associated to a single leaf node, namely, the leaf node obtained by routing the input vector from the root along the branches with higher probability. SRTs exhibit the conditional computational property, i.e., each prediction depends on a small number of nodes (parameters), and our nonlinear optimization formulation for training them is amenable to decomposition. After showing a universal approximation result for SRTs, we present a decomposition training algorithm including a clustering-based initialization procedure and a heuristic for reassigning the input vectors along the tree. Under mild assumptions, we establish asymptotic convergence guarantees. Experiments on 15 wellknown datasets indicate that our SRTs and decomposition algorithm yield higher accuracy and robustness compared with traditional soft regression trees trained using the nonlinear optimization formulation of Blanquero et al., and a significant reduction in training times as well as a slightly better average accuracy compared with the mixed-integer optimization approach of Bertsimas and Dunn. We also report a comparison with the Random Forest ensemble method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑