Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data
Inria Paris（法国国家信息与自动化研究所巴黎分部）；Inria Saclay（法国国家信息与自动化研究所萨克雷分部）；Ecole Normale Superieure PSL University（巴黎高等师范学院PSL大学）；University of Stuttgart（斯图加特大学）
David Holzmüller, Léo Grinsztajn, Ingo Steinwart
备注：
NeurIPS 2024. Changes in v3: mention bug in XGBoost results, mention original name of he+5 method. Code is available at github.com/dholzmueller/pytabkit
For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K--500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.
University of Freiburg（弗莱堡大学）；Leibniz University Hannover（莱布尼茨汉诺威大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using …
University of Maryland（马里兰大学）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）
Boosting is a method for finding a highly accurate hypothesis by linearly combining many ``weak" hypotheses, each of which may be only moderately accurate. Thus, boosting is a method for learning an ensemble of classifiers. While boosting has been shown to be very effective for decision trees, its impact on neural networks has not been extensively studied. We prove one important difference between sums of decision trees compared to sums of convolutional neural networks (CNNs) which is that a sum of decision trees cannot be represented by a single decision tree with the same number of parameter…
Stanford University（斯坦福大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of California, Berkeley（加州大学伯克利分校）；École Polytechnique（巴黎高等理工学院）
Is the Statistical Advantage Worth the Cost? An Empirical Comparison of KANs and MLPs for Structured Data Classification
Institute of Computer Science, University of the Philippines Los Baños（菲律宾大学洛斯巴尼奥斯分校计算机科学研究所）；Machine Learning and Artificial Intelligence Applications Lab, University of the Philippines Los Baños（菲律宾大学洛斯巴尼奥斯分校机器学习与人工智能应用实验室）
A Closer Look at Deep Learning Methods on Tabular Datasets
School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学）
Neural networks have proved to be very robust at processing unstructured data like images, text, videos, and audio. However, it has been observed that their performance is not up to the mark in tabular data; hence tree-based models are preferred in such scenarios. A popular model for tabular data is boosted trees, a highly efficacious and extensively used machine learning method, and it also provides good interpretability compared to neural networks. In this paper, we describe a novel architecture XBNet, which tries to combine tree-based models with that of neural networks to create a robust a…
University of Maryland（马里兰大学）；Amazon（亚马逊）；Shanghai Jiao Tong University（上海交通大学）
For supervised learning with tabular data, decision tree ensembles produced via boosting techniques generally dominate real-world applications involving iid training/test sets. However for graph data where the iid assumption is violated due to structured relations between samples, it remains unclear how to best incorporate this structure within existing boosting pipelines. To this end, we propose a generalized framework for iterating boosting with graph propagation steps that share node/sample information across edges connecting related samples. Unlike previous efforts to integrate graph-based…
Imitation learning for clinical decision support in pediatric ECMO
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Technische Universität Darmstadt（德累斯顿技术大学）；Swarthmore College（斯沃斯里学院）；University of Texas Southwestern Medical Center（德克萨斯西南医学中心）
There is growing interest in neural network architectures for tabular data. Many general-purpose tabular deep learning models have been introduced recently, with performance sometimes rivaling gradient boosted decision trees (GBDTs). These recent models draw inspiration from various sources, including GBDTs, factorization machines, and neural networks from other application domains. Previous tabular neural networks are also drawn upon, but are possibly under-considered, especially models associated with specific tabular problems. This paper focuses on several such models, and proposes modifica…
Duke University（杜克大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Though recent works have developed methods that can generate estimates (or imputations) of the missing entries in a dataset to facilitate downstream analysis, most depend on assumptions that may not align with real-world applications and could suffer from poor performance in subsequent tasks such as classification. This is particularly true if the data have large missingness rates or a small sample size. More importantly, the imputation error could be propagated into the prediction step that follows, which may constrain the capabilities of the prediction model. In this work, we introduce the g…
Gradient Boosted Decision Trees (GBDTs) are dominant machine learning algorithms for modeling discrete or tabular data. Unlike neural networks with millions of trainable parameters, GBDTs optimize loss function in an additive manner and have a single trainable parameter per leaf, which makes it easy to apply high-order optimization of the loss function. In this paper, we introduce high-order optimization for GBDTs based on numerical optimization theory which allows us to construct trees based on high-order derivatives of a given loss function. In the experiments, we show that high-order optimi…
Continual Contrastive Learning on Tabular Data with Out of Distribution
Indonesia Endowment Fund for Education Agency (LPDP)（印度尼西亚教育基金机构（LPDP））；School of Electrical Engineering and Computer Science（电气工程与计算机科学学院）；The University of Queensland（昆士兰大学）；Department of Computing（计算机系）；The Hong Kong Polytechnic University（香港理工大学）
Gradient-boosted regression trees (GBRTs) are hugely popular for solving tabular regression problems, but provide no estimate of uncertainty. We propose Instance-Based Uncertainty estimation for Gradient-boosted regression trees (IBUG), a simple method for extending any GBRT point predictor to produce probabilistic predictions. IBUG computes a non-parametric distribution around a prediction using the $k$-nearest training instances, where distance is measured with a tree-ensemble kernel. The runtime of IBUG depends on the number of training examples at each leaf in the ensemble, and can be impr…
Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard
School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学）；National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）
University of Tübingen（蒂宾根大学）；Schufa Holding AG（Schufa控股公司）
Heterogeneous tabular data are the most commonly used form of data and are essential for numerous critical and computationally demanding applications. On homogeneous data sets, deep neural networks have repeatedly shown excellent performance and have therefore been widely adopted. However, their adaptation to tabular data for inference or data generation tasks remains challenging. To facilitate further progress in the field, this work provides an overview of state-of-the-art deep learning methods for tabular data. We categorize these methods into three groups: data transformations, specialized…
University of Tuebingen（图宾根大学）；Amazon（亚马逊）；Deekard（迪卡德）
Gradient boosting machines (GBMs) based on decision trees consistently demonstrate state-of-the-art results on regression and classification tasks with tabular data, often outperforming deep neural networks. However, these models do not provide well-calibrated predictive uncertainties, which prevents their use for decision making in high-risk applications. The Bayesian treatment is known to improve predictive uncertainty calibration, but previously proposed Bayesian GBM methods are either computationally expensive, or resort to crude approximations. Variational inference is often used to imple…
