Tabular data is the foundation of the information age and has been extensively studied. Recent studies show that neural-based models are effective in learning contextual representation for tabular data. The learning of an effective contextual representation requires meaningful features and a large amount of data. However, current methods often fail to properly learn a contextual representation from the features without semantic information. In addition, it's intractable to enlarge the training set through mixed tabular datasets due to the difference between datasets. To address these problems, we propose a novel framework PTab, using the Pre-trained language model to model Tabular data. PTab learns a contextual representation of tabular data through a three-stage processing: Modality Transformation(MT), Masked-Language Fine-tuning(MF), and Classification Fine-tuning(CF). We initialize our model with a pre-trained Model (PTM) which contains semantic information learned from the large-scale language data. Consequently, contextual representation can be learned effectively during the fine-tuning stages. In addition, we can naturally mix the textualized tabular data to enlarge the training set to further improve representation learning. We evaluate PTab on eight popular tabular classification datasets. Experimental results show that our method has achieved a better average AUC score in supervised settings compared to the state-of-the-art baselines(e.g. XGBoost), and outperforms counterpart methods under semi-supervised settings. We present visualization results that show PTab has well instance-based interpretability.
Tabular biomedical data is often high-dimensional but with a very small number of samples. Although recent work showed that well-regularised simple neural networks could outperform more sophisticated architectures on tabular data, they are still prone to overfitting on tiny datasets with many potentially irrelevant features. To combat these issues, we propose Weight Predictor Network with Feature Selection (WPFS) for learning neural networks from high-dimensional and small sample data by reducing the number of learnable parameters and simultaneously performing feature selection. In addition to…
University of Maryland, College Park（马里兰大学帕克分校）；Capital One（第一资本金融公司）
Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-superv…
University of Cambridge（剑桥大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Alan Turing Institute（艾伦·图灵研究所）
Despite their success with unstructured data, deep neural networks are not yet a panacea for structured tabular data. In the tabular domain, their efficiency crucially relies on various forms of regularization to prevent overfitting and provide strong generalization performance. Existing regularization techniques include broad modelling decisions such as choice of architecture, loss functions, and optimization methods. In this work, we introduce Tabular Neural Gradient Orthogonalization and Specialization (TANGOS), a novel framework for regularization in the tabular setting built on latent uni…
ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data
Lane Dept. of Computer Science and Electrical Engineering（兰恩计算机科学与电气工程系）；Dept. of Geology & Geography（地质与地理系）；West Virginia University（西弗吉尼亚大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Green University of Bangladesh（孟加拉国绿色大学）；Stamford University Bangladesh（班加罗尔斯坦福大学）
Representation Learning for Tabular Data: A Comprehensive Survey
School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学）
Continual Contrastive Learning on Tabular Data with Out of Distribution
Indonesia Endowment Fund for Education Agency (LPDP)（印度尼西亚教育基金机构（LPDP））；School of Electrical Engineering and Computer Science（电气工程与计算机科学学院）；The University of Queensland（昆士兰大学）；Department of Computing（计算机系）；The Hong Kong Polytechnic University（香港理工大学）
Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）；Leibniz Institute for Prevention Research and Epidemiology – BIPS（莱比锡预防研究与流行病学研究所 – BIPS）；Faculty of Mathematics and Computer Science, University of Bremen（不莱梅大学数学与计算机科学学院）；Faculty of Physics and Electrical Engineering, University of Bremen（不莱梅大学物理与电气工程学院）
TabFORGE: A Tabular Foundation Model for Generative Modelling
Department of Computer Science and Technology, University of Cambridge, UK（计算机科学与技术系，剑桥大学，英国）；SAP SE；Télécom Paris, Institut Polytechnique de Paris, France（巴黎telecom，巴黎理工 institute，法国）
While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art …
Stanford University（斯坦福大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of California, Berkeley（加州大学伯克利分校）；École Polytechnique（巴黎高等理工学院）
Tabular data synthesis has received wide attention in the literature. This is because available data is often limited, incomplete, or cannot be obtained easily, and data privacy is becoming increasingly important. In this work, we present a generalized GAN framework for tabular synthesis, which combines the adversarial training of GANs and the negative log-density regularization of invertible neural networks. The proposed framework can be used for two distinctive objectives. First, we can further improve the synthesis quality, by decreasing the negative log-density of real records in the proce…
Preserving logical and functional dependencies in synthetic tabular data
Institute of Computer Science, University of Rostock（罗斯托克大学计算机科学学院）；Leibniz-Institute for Food Systems Biology（莱布尼茨食品系统生物学研究所）；Technical University of Munich, Freising（慕尼黑工业大学弗赖辛校区）；Stellenbosch Institute for Advanced Study（斯泰伦博斯高级研究院）；Indian Institute of Science Education and Research, Thiruvananthapuram（特里凡得琅印度科学教育与研究学院）
TabGift: A Context-Efficient Foundation Model for Tabular Data Generation
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Score-based generative models (SGMs) are a recent breakthrough in generating fake images. SGMs are known to surpass other generative models, e.g., generative adversarial networks (GANs) and variational autoencoders (VAEs). Being inspired by their big success, in this work, we fully customize them for generating fake tabular data. In particular, we are interested in oversampling minor classes since imbalanced classes frequently lead to sub-optimal training outcomes. To our knowledge, we are the first presenting a score-based tabular data oversampling method. Firstly, we re-design our own score …
Beyond IID: How General Are Tabular Foundation Models, Really?
Prior Labs；University of Freiburg（弗赖堡大学）；University of Mannheim（曼海姆大学）；INRIA Saclay（法国国家信息与自动化研究所萨克雷研究中心）；Technion（技术学院）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Zuse School ELIZA（Zuse学校ELIZA）
MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinement
GE Healthcare（GE医疗）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carle Illinois College of Medicine, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校卡尔伊利诺伊医学院）
IBM Research（IBM研究院）；T.J Watson Research Center（T.J沃森研究中心）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Tabular datasets are ubiquitous in data science applications. Given their importance, it seems natural to apply state-of-the-art deep learning algorithms in order to fully unlock their potential. Here we propose neural network models that represent tabular time series that can optionally leverage their hierarchical structure. This results in two architectures for tabular time series: one for learning representations that is analogous to BERT and can be pre-trained end-to-end and used in downstream tasks, and one that is akin to GPT and can be used for generation of realistic synthetic tabular …
Lomonosov Moscow State University（莫斯科国立大学（罗蒙诺索夫莫斯科国立大学））
Data distillation is the problem of reducing the volume oftraining data while keeping only the necessary information. With thispaper, we deeper explore the new data distillation algorithm, previouslydesigned for image data. Our experiments with tabular data show thatthe model trained on distilled samples can outperform the model trainedon the original dataset. One of the problems of the considered algorithmis that produced data has poor generalization on models with differenthyperparameters. We show that using multiple architectures during distillation can help overcome this problem.
Given a labeled training set and a collection of unlabeled data, the goal of active learning (AL) is to identify the best unlabeled points to label. In this comprehensive study, we analyze the performance of a variety of AL algorithms on deep neural networks trained on 69 real-world tabular classification datasets from the OpenML-CC18 benchmark. We consider different data regimes and the effect of self-supervised model pre-training. Surprisingly, we find that the classical margin sampling technique matches or outperforms all others, including current state-of-art, in a wide range of experiment…
TabularQGAN: A quantum generative model for tabular data synthesis
SAP SE（SAP公司）；BASF Digital Solutions（巴斯夫数字解决方案）；QUTAC Quantum Technology and Application Consortium（QUTAC量子技术与应用联盟）；Technical University of Munich CIT（慕尼黑技术大学CIT）；Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学）；Merck KGaA（默克公司）
Fully connected deep neural networks (DNN) often include redundant weights leading to overfitting and high memory requirements. Additionally, the performance of DNN is often challenged by traditional machine learning models in tabular data classification. In this paper, we propose periodical perturbations (prune and regrow) of DNN weights, especially at the self-supervised pre-training stage of deep autoencoders. The proposed weight perturbation strategy outperforms dropout learning in four out of six tabular data sets in downstream classification tasks. The L1 or L2 regularization of weights …
