首页/论文/2007.04446

arXiv 2007.044462020-07-08stat.MLcs.LGstat.APstat.CO

StructureBoost: Efficient Gradient Boosting for Structured Categorical Variables

Numeristical

Brian Lucena

英文摘要：

Gradient boosting methods based on Structured Categorical Decision Trees (SCDT) have been demonstrated to outperform numerical and one-hot-encodings on problems where the categorical variable has a known underlying structure. However, the enumeration procedure in the SCDT is infeasible except for categorical variables with low or moderate cardinality. We propose and implement two methods to overcome the computational obstacles and efficiently perform Gradient Boosting on complex structured categorical variables. The resulting package, called StructureBoost, is shown to outperform established packages such as CatBoost and LightGBM on problems with categorical predictors that contain sophisticated structure. Moreover, we demonstrate that StructureBoost can make accurate predictions on unseen categorical values due to its knowledge of the underlying structure.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

CatBoost: gradient boosting with categorical features support

Yandex（扬德克斯）

In this paper we present CatBoost, a new open-sourced gradient boosting library that successfully handles categorical features and outperforms existing publicly available implementations of gradient boosting in terms of quality on a set of popular publicly available datasets. The library has a GPU implementation of learning algorithm and a CPU implementation of scoring algorithm, which are significantly faster than other gradient boosting libraries on ensembles of similar sizes.

2018-10-24cs.LGcs.MSstat.ML

针对具有高基数分类变量的数据的机器学习方法比较

A Comparison of Machine Learning Methods for Data with High-Cardinality Categorical Variables

Lucerne University of Applied Sciences and Arts（卢塞恩应用科学与艺术大学）

本文针对高基数分类变量问题，实证比较树提升、深度神经网络与线性混合效应模型，发现带随机效应的机器学习模型预测准确率更高，且带随机效应的树提升优于带随机效应的深度神经网络。

2023-07-05cs.AIcs.LGstat.ML

从点预测到概率梯度提升：用于索赔频率与严重性预测

From Point to probabilistic gradient boosting for claim frequency and severity prediction

École d’actuariat, Université Laval（actuary school, Laval University）

本文统一对比了十种点预测与概率梯度提升算法在五个公开索赔数据集上的表现，发现LightGBM和XGBoostLSS计算效率最优，CatBoost在高基数分类变量下预测更佳，且模型充分性与预测准确性可兼得。

2025-04-28cs.LGstat.ML

Wasserstein梯度提升：一种分布值监督学习框架

Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised Learning

The University of Edinburgh（爱丁堡大学）

本文提出Wasserstein梯度提升框架，将梯度提升扩展到分布值监督学习，通过拟合Wasserstein梯度伪残差实现基于树的证据学习，并在概率预测上优于现有不确定性量化方法。

2024-08-29cs.LGstat.MEstat.ML

基于路径的梯度提升用于图级预测

Path-Based Gradient Boosting for Graph-Level Prediction

Department of Mathematics, University of Oslo（奥斯陆大学数学系）

本文提出PathBoost，一种基于图结构直接学习判别路径特征的梯度树提升方法，通过逻辑损失实现二分类，结合节点和边属性，自动选择锚点节点，提升图级预测性能。

2026-04-26cs.LGstat.ML

Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection

Tel Aviv University（特拉维夫大学）

Gradient Boosting Machines (GBM) are among the go-to algorithms on tabular data, which produce state of the art results in many prediction tasks. Despite its popularity, the GBM framework suffers from a fundamental flaw in its base learners. Specifically, most implementations utilize decision trees that are typically biased towards categorical variables with large cardinalities. The effect of this bias was extensively studied over the years, mostly in terms of predictive performance. In this work, we extend the scope and study the effect of biased base learners on GBM feature importance (FI) m…

2021-09-12cs.LGstat.ML

更多相关论文收起更多论文

Automatic Gradient Boosting

LMU（慕尼黑大学）

Automatic machine learning performs predictive modeling with high performing machine learning tools without human interference. This is achieved by making machine learning applications parameter-free, i.e. only a dataset is provided while the complete model selection and model building process is handled internally through (often meta) optimization. Projects like Auto-WEKA and auto-sklearn aim to solve the Combined Algorithm Selection and Hyperparameter optimization (CASH) problem resulting in huge configuration spaces. However, for most real-world applications, the optimization over only a fe…

2018-07-13cs.LGstat.ML

Gradient Boosting on Decision Trees for Mortality Prediction in Transcatheter Aortic Valve Implantation

Eindhoven University of Technology（埃因霍温理工大学）；Catharina Hospital（卡特琳娜医院）

Current prognostic risk scores in cardiac surgery are based on statistics and do not yet benefit from machine learning. Statistical predictors are not robust enough to correctly identify patients who would benefit from Transcatheter Aortic Valve Implantation (TAVI). This research aims to create a machine learning model to predict one-year mortality of a patient after TAVI. We adopt a modern gradient boosting on decision trees algorithm, specifically designed for categorical features. In combination with a recent technique for model interpretations, we developed a feature analysis and selection…

2020-01-08cs.CYcs.LGstat.ML

Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features

LMU Munich（慕尼黑大学）

Since most machine learning (ML) algorithms are designed for numerical inputs, efficiently encoding categorical variables is a crucial aspect in data analysis. A common problem are high cardinality features, i.e. unordered categorical predictor variables with a high number of levels. We study techniques that yield numeric representations of categorical variables which can then be used in subsequent ML applications. We focus on the impact of these techniques on a subsequent algorithm's predictive performance, and -- if possible -- derive best practices on when to use which technique. We conduct…

2022-03-04cs.LGstat.ML

LambdaOpt: Learn to Regularize Recommender Models in Finer Levels

Microsoft Research（微软研究院）

Recommendation models mainly deal with categorical variables, such as user/item ID and attributes. Besides the high-cardinality issue, the interactions among such categorical variables are usually long-tailed, with the head made up of highly frequent values and a long tail of rare ones. This phenomenon results in the data sparsity issue, making it essential to regularize the models to ensure generalization. The common practice is to employ grid search to manually tune regularization hyperparameters based on the validation data. However, it requires non-trivial efforts and large computation res…

2019-05-28cs.IRcs.LGstat.ML