There is an increasing interest in the application of deep learning architectures to tabular data. One of the state-of-the-art solutions is TabTransformer which incorporates an attention mechanism to better track relationships between categorical features and then makes use of a standard MLP to output its final logits. In this paper we propose multiple modifications to the original TabTransformer performing better on binary classification tasks for three separate datasets with more than 1% AUROC gains. Inspired by gated MLP, linear projections are implemented in the MLP block and multiple activation functions are tested. We also evaluate the importance of specific hyper parameters during training.
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Jiaotong University（北京交通大学）
Tabular data is the foundation of the information age and has been extensively studied. Recent studies show that neural-based models are effective in learning contextual representation for tabular data. The learning of an effective contextual representation requires meaningful features and a large amount of data. However, current methods often fail to properly learn a contextual representation from the features without semantic information. In addition, it's intractable to enlarge the training set through mixed tabular datasets due to the difference between datasets. To address these problems,…
Code and Pixels: Multi-Modal Contrastive Pre-training for Enhanced Tabular Data Analysis
Karolinska Institute（卡罗林斯卡学院）；Helmholtz Imaging（亥姆霍兹成像研究所）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Institute of Materials Systems Modeling（材料系统建模研究所）；Helmholtz-Zentrum Hereon（亥姆霍兹-中心赫伦）；Institute of Membrane Research（膜研究所）
University of Maryland, College Park（马里兰大学帕克分校）；Capital One（第一资本金融公司）
Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-superv…
Airt Research（Airt研究中心）；Algebra University College（代数大学学院）
Time-series classification is one of the most frequently performed tasks in industrial data science, and one of the most widely used data representation in the industrial setting is tabular representation. In this work, we propose a novel scalable architecture for learning representations from tabular time-series data and subsequently performing downstream tasks such as time-series classification. The representation learning framework is end-to-end, akin to bidirectional encoder representations from transformers (BERT) in language modeling, however, we introduce novel masking technique suitabl…
Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）；Leibniz Institute for Prevention Research and Epidemiology – BIPS（莱比锡预防研究与流行病学研究所 – BIPS）；Faculty of Mathematics and Computer Science, University of Bremen（不莱梅大学数学与计算机科学学院）；Faculty of Physics and Electrical Engineering, University of Bremen（不莱梅大学物理与电气工程学院）
Self-Improving Tabular Language Models via Iterative Reward-Guided Post-Training
Department of Engineering, University of Cambridge（剑桥大学工程系）；CISPA Helmholtz Center for Information Security, Saarbrücken, Germany（德国萨尔布吕肯信息安全中心）；The Alan Turing Institute, London（伦敦阿兰·图灵研究所）
University of the Witwatersrand（威特沃特斯兰德大学）；ETH Zurich（苏黎世联邦理工学院）
Deep learning models have gained great popularity in statistical modeling because they lead to very competitive regression models, often outperforming classical statistical models such as generalized linear models. The disadvantage of deep learning models is that their solutions are difficult to interpret and explain, and variable selection is not easily possible because deep learning models solve feature engineering and variable selection internally in a nontransparent way. Inspired by the appealing structure of generalized linear models, we propose a new network architecture that shares simi…
TabKAN: Advancing Tabular Data Analysis using Kolmogorov-Arnold Network
Department of Computer Science, University of Kentucky（卡内基梅隆大学计算机科学系）；Independent Researcher（独立研究者）；Institute for Biomedical Informatics, University of Kentucky（生物医学信息学研究所）
Korea University（高丽大学）；Korea Institute for Defense Analyses（韩国国防分析研究所）
Since demand is influenced by a wide variety of causes, it is necessary to decompose the explana-tory variables into different levels, extract their relationships effectively, and reflect them in the forecast. In particular, this contextual information can be very useful in demand forecasting with large demand volatility or intermittent demand patterns. Convolutional neural networks (CNNs) have been successfully used in many fields where important information in data is represented by images. CNNs are powerful because they accept samples as images and use adjacent voxel sets to integrate multi…
GeoViSTA: Geospatial Vision-Tabular Transformer for Multimodal Environment Representation
Department of Electrical and Computer Engineering, Rice University（理海大学电气与计算机工程系）；Center for Cardiovascular Computational and Precision Health, Department of Cardiology, DeBakey Heart and Vascular Center, Houston Methodist（休斯顿方法主义医疗中心心血管计算与精准健康中心、心内科部门、德贝基心脏和血管中心）
Prediction over tabular data is an essential and fundamental problem in many important downstream tasks. However, existing methods either take a data instance of the table independently as input or do not fully utilize the multi-rows features and labels to directly change and enhance the target data representations. In this paper, we propose to 1) construct a hypergraph from relevant data instance retrieval to model the cross-row and cross-column patterns of those instances, and 2) perform message Propagation to Enhance the target data instance representation for Tabular prediction tasks. Spec…
We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. Through extensive experiments on fifteen publicly available datasets, we show that the TabTransformer outperforms the state-of-the-art deep learning methods for tabular data by at least 1.0% on mean AUC, and matches the performance of tree-based ensemble models. Furt…
IBM Research（IBM研究院）；T.J Watson Research Center（T.J沃森研究中心）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Tabular datasets are ubiquitous in data science applications. Given their importance, it seems natural to apply state-of-the-art deep learning algorithms in order to fully unlock their potential. Here we propose neural network models that represent tabular time series that can optionally leverage their hierarchical structure. This results in two architectures for tabular time series: one for learning representations that is analogous to BERT and can be pre-trained end-to-end and used in downstream tasks, and one that is akin to GPT and can be used for generation of realistic synthetic tabular …
Explainable Dual-Attention Tabular Transformer for Soil Electrical Resistivity Prediction: A Decision Support Framework for High-Voltage Substation Construction
King Mongkut’s University of technology Thonburi（泰国国王科技大学）
本研究提出一种用于预测土壤电阻率的双注意力 Transformer 架构，结合粒子群优化调参，在保持高预测精度的同时通过 SHAP 值提供可解释性，并开发了辅助高压变电站建设的 Web 决策支持框架。
MIT LIDS（麻省理工学院信息与决策系统实验室）；MRC-BSU, University of Cambridge（剑桥大学MRC生物统计组）；Universidad Rey Juan Carlos（胡安·卡洛斯国王大学）
Modeling the probability distribution of rows in tabular data and generating realistic synthetic data is a non-trivial task. Tabular data usually contains a mix of discrete and continuous columns. Continuous columns may have multiple modes whereas discrete columns are sometimes imbalanced making the modeling difficult. Existing statistical and deep neural network models fail to properly model this type of data. We design TGAN, which uses a conditional generative adversarial network to address these challenges. To aid in a fair and thorough comparison, we design a benchmark with 7 simulated and…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；University of South Carolina（南卡罗来纳大学）
The vast majority of research on explainability focuses on post-explainability rather than explainable modeling. Namely, an explanation model is derived to explain a complex black box model built with the sole purpose of achieving the highest performance possible. In part, this trend might be driven by the misconception that there is a trade-off between explainability and accuracy. Furthermore, the consequential work on Shapely values, grounded in game theory, has also contributed to a new wave of post-explainability research on better approximations for various machine learning models, includ…
Rethinking Distribution Shifts: Empirical Analysis and Modeling for Tabular Data
Department of Industrial Engineering and Operations Research（工业工程与运筹学系）；Department of Computer Science and Technology（计算机科学与技术系）；Decision, Risk, and Operations Division（决策、风险与运营部）；Columbia University（哥伦比亚大学）；Tsinghua University（清华大学）
Structural Equation-VAE: Disentangled Latent Representations for Tabular Data
Department of Politics and Public Administration（政治与公共行政系）；The University of Hong Kong（香港大学）；School of Computer Science（计算机科学系）；Carnegie Mellon University（卡内基梅隆大学）；Department of Applied Social Sciences（应用社会科学系）；The Hong Kong Polytechnic University（香港理工大学）
Federated Diffusion Modeling with Differential Privacy for Tabular Data Synthesis
Deutsche Bundesbank（德国联邦银行）；University of St.Gallen（圣加尔大学）；Swiss Federal Audit Office（瑞士联邦审计局）；International Computer Science Institute (ICSI)（国际计算机科学研究所）
Accurate and Robust Generative Approach for Overcoming Data Sparsity and Imbalance in Landslide Modeling with A Tabular Foundation Model
School of Engineering and Technology, China University of Geosciences (Beijing)（工程与技术学院，中国地质大学（北京））；School of Geological Engineering and Geomatics, Chang'an University（地质工程与测绘学院，长安大学）
Institute of Transportation Studies（交通研究学院）；University of California, Davis（加州大学戴维斯分校）
Convolutional Neural Networks (CNN) have been a good solution for understanding a vast image dataset. As the increased number of battery-equipped electric vehicles is flourishing globally, there has been much research on understanding which charge levels electric vehicle drivers would choose to charge their vehicles to get to their destination without any prevention. We implemented deep learning approaches to analyze the tabular datasets to understand their state of charge and which charge levels they would choose. In addition, we implemented the Image Generator for Tabular Dataset algorithm t…
HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling
Ben Gurion University of the Negev（本· Gurion 大学）；Centre for Intelligent Machines, McGill University（智能机器中心，麦吉尔大学）；Mila - Quebec AI Institute（魁北克人工智能研究所）
