We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. Through extensive experiments on fifteen publicly available datasets, we show that the TabTransformer outperforms the state-of-the-art deep learning methods for tabular data by at least 1.0% on mean AUC, and matches the performance of tree-based ensemble models. Furthermore, we demonstrate that the contextual embeddings learned from TabTransformer are highly robust against both missing and noisy data features, and provide better interpretability. Lastly, for the semi-supervised setting we develop an unsupervised pre-training procedure to learn data-driven contextual embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art methods.
There is an increasing interest in the application of deep learning architectures to tabular data. One of the state-of-the-art solutions is TabTransformer which incorporates an attention mechanism to better track relationships between categorical features and then makes use of a standard MLP to output its final logits. In this paper we propose multiple modifications to the original TabTransformer performing better on binary classification tasks for three separate datasets with more than 1% AUROC gains. Inspired by gated MLP, linear projections are implemented in the MLP block and multiple acti…
Application of Tabular Transformer Architectures for Operating System Fingerprinting
RNASNA-IMEDIR Research Group Department of Computer Science and Information Technologies Facultad de Informática Universidade da Coruña（RNASNA-IMEDIR研究组计算机科学与信息科技系信息学院科鲁纳大学）；CITIC Research Centre Universidade da Coruña（CITIC研究中心科鲁纳大学）；IKERDATA S.L（IKERDATA公司）
TabAttackBench: A Benchmark for Adversarial Attacks on Tabular Data
School of Information Systems, Queensland University of Technology（昆士兰理工大学信息系统学院）；Center for Data Science, Queensland University of Technology（昆士兰理工大学数据科学中心）；School of Software, Tsinghua University（清华大学软件学院）；NOVA Information Management School, NOVA University of Lisbon（里斯本大学NOVA信息管理学院）；Data Science Institute, University of Technology Sydney（悉尼科技大学数据科学研究院）；INESC-ID/Instituto Superior Técnico, University of Lisboa（里斯本大学INESC-ID/技术高等学院）
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Jiaotong University（北京交通大学）
Tabular data is the foundation of the information age and has been extensively studied. Recent studies show that neural-based models are effective in learning contextual representation for tabular data. The learning of an effective contextual representation requires meaningful features and a large amount of data. However, current methods often fail to properly learn a contextual representation from the features without semantic information. In addition, it's intractable to enlarge the training set through mixed tabular datasets due to the difference between datasets. To address these problems,…
Gradient Boosting Decision Trees on Medical Diagnosis over Tabular Data
Department of Electrical and Computer Engineering（电气与计算机工程系）；Northeastern University（东北大学）；The Business School（商学院）；Worcester Polytechnic Institute（沃思利理工学院）
CTG-KrEW: Generating Synthetic Structured Contextually Correlated Content by Conditional Tabular GAN with K-Means Clustering and Efficient Word Embedding
Indian Institute of Technology Kharagpur（印度理工学院克勒格布尔分校）；Missouri University of Science and Technology（密苏里科技大学）
TabText: Language-Based Representations of Tabular Health Data for Predictive Modelling
Tandon School of Engineering（坦顿工程学院）；New York University（纽约大学）；Operations Research Center（运筹学中心）；Massachusetts Institute of Technology（麻省理工学院）；University of Wisconsin, Madison（威斯康星大学麦迪逊分校）；Foster School of Business（福斯特商学院）；University of Washington, Foster School of Business（华盛顿大学福斯特商学院）；Stern School of Business（斯特恩商学院）；Abdul Latif Jameel Clinic for Machine Learning in Health（阿卜杜勒拉赫曼·贾迈尔健康机器学习诊所）；Sloan School of Management（斯隆管理学院）
Portland State University（波特兰州立大学）；Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；University of Chinese Academy of Sciences（中国科学院大学）；Dalian Maritime University（大连海事大学）；Jilin University（吉林大学）
We present BRIDGE, a powerful sequential architecture for modeling dependencies between natural language questions and relational databases in cross-DB semantic parsing. BRIDGE represents the question and DB schema in a tagged sequence where a subset of the fields are augmented with cell values mentioned in the question. The hybrid sequence is encoded by BERT with minimal subsequent layers and the text-DB contextualization is realized via the fine-tuned deep attention in BERT. Combined with a pointer-generator decoder with schema-consistency driven search space pruning, BRIDGE attained state-o…
University of Oxford（牛津大学）；The Alan Turing Institute（艾伦·图灵研究所）；University of Oslo（奥斯陆大学）；The University of Edinburgh（爱丁堡大学）
The usefulness of tabular data such as web tables critically depends on understanding their semantics. This study focuses on column type prediction for tables without any meta data. Unlike traditional lexical matching-based methods, we propose a deep prediction model that can fully exploit a table's contextual semantics, including table locality features learned by a Hybrid Neural Network (HNN), and inter-column semantics features learned by a knowledge base (KB) lookup and query answering algorithm.It exhibits good performance not only on individual table sets, but also when transferring from…
Korea University（高丽大学）；Korea Institute for Defense Analyses（韩国国防分析研究所）
Since demand is influenced by a wide variety of causes, it is necessary to decompose the explana-tory variables into different levels, extract their relationships effectively, and reflect them in the forecast. In particular, this contextual information can be very useful in demand forecasting with large demand volatility or intermittent demand patterns. Convolutional neural networks (CNNs) have been successfully used in many fields where important information in data is represented by images. CNNs are powerful because they accept samples as images and use adjacent voxel sets to integrate multi…
DALL-M: Context-Aware Clinical Data Augmentation with LLMs
School of Information Systems, Queensland University of Technology（昆士兰科技大学信息系统学院）；School of Computer Science, Queensland University of Technology（昆士兰科技大学计算机科学学院）；Data Science Institute, University of Technology Sydney（悉尼科技大学数据科学研究所）；Imagiology Department, Grupo Lusiadas（卢西塔达斯集团影像学系）
