Despite much work on advanced deep learning and generative modeling techniques for tabular data generation and imputation, traditional methods have continued to win on imputation benchmarks. We herein present UnmaskingTrees, a simple method for tabular imputation (and generation) employing gradient-boosted decision trees which are used to incrementally unmask individual features. On a benchmark for out-of-the-box performance on 27 small tabular datasets, UnmaskingTrees offers leading performance on imputation; state-of-the-art performance on generation given data with missingness; and competitive performance on vanilla generation given data without missingness. To solve the conditional generation subproblem, we propose a tabular probabilistic prediction method, BaltoBot, which fits a balanced tree of boosted tree classifiers. Unlike older methods, it requires no parametric assumption on the conditional distribution, accommodating features with multimodal distributions; unlike newer diffusion methods, it offers fast sampling, closed-form density estimation, and flexible handling of discrete variables. We finally consider our two approaches as meta-algorithms, demonstrating in-context learning-based generative modeling with TabPFN.
Unmasking the Chameleons: A Benchmark for Out-of-Distribution Detection in Medical Tabular Data
Amsterdam Public Health Research Institute（阿姆斯特丹公共卫生研究院）；Amsterdam UMC（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）；Institute of Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）；Pacmed（Pacmed公司）
Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression
Department of Statistics, Federal University of São Carlos (UFSCar)（圣卡洛斯联邦大学统计系）；Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学、法国国家信息与自动化研究所、法国国家科学研究中心、格勒诺布尔国立理工学院、让·库尔坦数学实验室）
Xiamen University（厦门大学）；Department of Electronic Science（电子科学系）
Differentiable forest is an ensemble of decision trees with full differentiability. Its simple tree structure is easy to use and explain. With full differentiability, it would be trained in the end-to-end learning framework with gradient-based optimization method. In this paper, we propose tree attention block(TAB) in the framework of differentiable forest. TAB block has two operations, squeeze and regulate. The squeeze operation would extract the characteristic of each tree. The regulate operation would learn nonlinear relations between these trees. So TAB block would learn the importance of …
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；CSIRO’s Data61（CSIRO数据61）；Monash University（墨尔本大学）；International Center of Future Science, Jilin University（吉林大学未来科学国际中心）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（知识驱动人机智能工程研究中心，教育部，中国）
Yandex（Yandex（央捷科斯））；Lomonosov Moscow State University（罗蒙诺索夫莫斯科国立大学）
Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learnin…
While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art …
RO-FIGS: Efficient and Expressive Tree-Based Ensembles for Tabular Data
Department of Computer Science and Technology, University of Cambridge（计算机科学与技术系，剑桥大学）；U.S. Army Medical Research and Development Command（美国陆军医疗研究与开发指挥部）
Deep neural networks have been proven powerful at processing perceptual data, such as images and audio. However for tabular data, tree-based models are more popular. A nice property of tree-based models is their natural interpretability. In this work, we present Deep Neural Decision Trees (DNDT) -- tree models realised by neural networks. A DNDT is intrinsically interpretable, as it is a tree. Yet as it is also a neural network (NN), it can be easily implemented in NN toolkits, and trained with gradient descent rather than greedy splitting. We evaluate DNDT on several tabular datasets, verify …
Department of Electronic Science, Xiamen University（厦门大学电子科学系）
We present a general architecture of deep differentiable forest and its sparse attention mechanism. The differentiable forest has the advantages of both trees and neural networks. Its structure is a simple binary tree, easy to use and understand. It has full differentiability and all variables are learnable parameters. We would train it by the gradient-based optimization method, which shows great power in the training of deep CNN. We find and analyze the attention mechanism in the differentiable forest. That is, each decision depends on only a few important features, and others are irrelevant.…
Amazon Web Services（亚马逊云科技）；University of Pennsylvania（宾夕法尼亚大学）
Automated machine learning (AutoML) can produce complex model ensembles by stacking, bagging, and boosting many individual models like trees, deep networks, and nearest neighbor estimators. While highly accurate, the resulting predictors are large, slow, and opaque as compared to their constituents. To improve the deployment of AutoML on tabular data, we propose FAST-DAD to distill arbitrarily complex ensemble predictors into individual models like boosted trees, random forests, and deep networks. At the heart of our approach is a data augmentation strategy based on Gibbs sampling from a self-…
We propose a novel high-performance and interpretable canonical deep tabular data learning architecture, TabNet. TabNet uses sequential attention to choose which features to reason from at each decision step, enabling interpretability and more efficient learning as the learning capacity is used for the most salient features. We demonstrate that TabNet outperforms other neural network and decision tree variants on a wide range of non-performance-saturated tabular datasets and yields interpretable feature attributions plus insights into the global model behavior. Finally, for the first time to o…
Zhejiang University（浙江大学）；Zhejiang University - University of Illinois at Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院）
Class imbalance has a detrimental effect on the predictive performance of most supervised learning algorithms as the imbalanced distribution can lead to a bias preferring the majority class. To solve this problem, we propose a Supervised Contrastive Learning (SCL) method with Tree-structured Parzen Estimator (TPE) technique for imbalanced tabular datasets. Contrastive learning (CL) can extract the information hidden in data even without labels and has shown some potential for imbalanced learning tasks. SCL further considers the label information based on CL, which also addresses the insufficie…
xRFM: Accurate, scalable, and interpretable feature learning models for tabular data
Computer Science and Engineering（计算机科学与工程）；INRIA Saclay；Halıcıoğlu Data Science Institute（Halıcıoğlu数据科学研究所）；UC San Diego（加州大学圣地亚哥分校）；MIT Mathematics（麻省理工学院数学系）；Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学博德研究所）
Boston University（波士顿大学）；Sorbonne Université（索邦大学）；Ecole Polytechnique（巴黎综合理工学院）
Dedicated neural network (NN) architectures have been designed to handle specific data types (such as CNN for images or RNN for text), which ranks them among state-of-the-art methods for dealing with these data. Unfortunately, no architecture has been found for dealing with tabular data yet, for which tree ensemble methods (tree boosting, random forests) usually show the best predictive performances. In this work, we propose a new sparse initialization technique for (potentially deep) multilayer perceptrons (MLP): we first train a tree-based procedure to detect feature interactions and use the…
Gradient Boosting Decision Trees on Medical Diagnosis over Tabular Data
Department of Electrical and Computer Engineering（电气与计算机工程系）；Northeastern University（东北大学）；The Business School（商学院）；Worcester Polytechnic Institute（沃思利理工学院）
Stanford University（斯坦福大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of California, Berkeley（加州大学伯克利分校）；École Polytechnique（巴黎高等理工学院）
Neural networks have proved to be very robust at processing unstructured data like images, text, videos, and audio. However, it has been observed that their performance is not up to the mark in tabular data; hence tree-based models are preferred in such scenarios. A popular model for tabular data is boosted trees, a highly efficacious and extensively used machine learning method, and it also provides good interpretability compared to neural networks. In this paper, we describe a novel architecture XBNet, which tries to combine tree-based models with that of neural networks to create a robust a…
Contrastive self-supervised learning has been successfully used in many domains, such as images, texts, graphs, etc., to learn features without requiring label information. In this paper, we propose a new local contrastive feature learning (LoCL) framework, and our theme is to learn local patterns/features from tabular data. In order to create a niche for local learning, we use feature correlations to create a maximum-spanning tree, and break the tree into feature subsets, with strongly correlated features being assigned next to each other. Convolutional learning of the features is used to lea…
TVineSynth: A Truncated C-Vine Copula Generator of Synthetic Tabular Data to Balance Privacy and Utility
Norwegian Computing Center（挪威计算中心）；University of Oslo（奥斯陆大学）；Technical University of Munich（慕尼黑工业大学）；Munich Data Science Institute（慕尼黑数据科学研究所）；Integreat - Norwegian Centre for Knowledge-driven Machine Learning（Integreat——挪威知识驱动型机器学习中心）
XGenBoost: Synthesizing Small and Large Tabular Datasets with XGBoost
Public Health \& Primary Care, Leiden University Medical Center, Leiden, The Netherlands；Research；Development, Statistics Netherlands (CBS), The Hague, The Netherlands；Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands
Tabular datasets with low-sample-size or many variables are prevalent in biomedicine. Practitioners in this domain prefer linear or tree-based models over neural networks since the latter are harder to interpret and tend to overfit when applied to tabular datasets. To address these neural networks' shortcomings, we propose an intrinsically interpretable network for heterogeneous biomedical data. We design a locally sparse neural network where the local sparsity is learned to identify the subset of most relevant features for each sample. This sample-specific sparsity is predicted via a \textit{…
Peter the Great St.Petersburg Polytechnic University（彼得大帝圣彼得堡理工大学）
A new random forest based model for solving the Multiple Instance Learning (MIL) problem under small tabular data, called Soft Tree Ensemble MIL (STE-MIL), is proposed. A new type of soft decision trees is considered, which is similar to the well-known soft oblique trees, but with a smaller number of trainable parameters. In order to train the trees, it is proposed to convert them into neural networks of a specific form, which approximate the tree functions. It is also proposed to aggregate the instance and bag embeddings (output vectors) by using the attention mechanism. The whole STE-MIL mod…
Paradigm Inc.（第四范式公司）；Hong Kong University of Science and Technology（香港科技大学）
Feature crossing captures interactions among categorical features and is useful to enhance learning from tabular data in real-world businesses. In this paper, we present AutoCross, an automatic feature crossing tool provided by 4Paradigm to its customers, ranging from banks, hospitals, to Internet corporations. By performing beam search in a tree-structured space, AutoCross enables efficient generation of high-order cross features, which is not yet visited by existing works. Additionally, we propose successive mini-batch gradient descent and multi-granularity discretization to further improve …
Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Data
Department of Statistics, Texas A\&M University, College Station, TX（德克萨斯A&M大学统计学系）；Department of Electrical and Computer Engineering, University of Minnesota, Twin City, MN（明尼苏达大学电气与计算机工程系）
Manipal Institute of Technology（马尼帕尔理工学院）；IBM Research（IBM研究院）
Explainability techniques for Graph Neural Networks still have a long way to go compared to explanations available for both neural and decision decision tree-based models trained on tabular data. Using a task that straddles both graphs and tabular data, namely Entity Matching, we comment on key aspects of explainability that are missing in GNN model explanations.
A Closer Look at Deep Learning Methods on Tabular Datasets
School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学）
