In recent years, significant efforts have been directed toward adapting modern neural network architectures for tabular data. However, despite their larger number of parameters and longer training and inference times, these models often fail to consistently outperform vanilla multilayer perceptron (MLP) neural networks. Moreover, MLP-based ensembles have recently demonstrated superior performance and efficiency compared to advanced deep learning methods. Therefore, rather than focusing on building deeper and more complex deep learning models, we propose investigating whether MLP neural networks can be replaced with more efficient architectures without sacrificing performance. In this paper, we first introduce GG MoE, a mixture-of-experts (MoE) model with a Gumbel-Softmax gating function. We then demonstrate that GG MoE with an embedding layer achieves the highest performance across $38$ datasets compared to standard MoE and MLP models. Finally, we show that both MoE and GG MoE utilize significantly fewer parameters than MLPs, making them a promising alternative for scaling and ensemble methods.
There is an increasing interest in the application of deep learning architectures to tabular data. One of the state-of-the-art solutions is TabTransformer which incorporates an attention mechanism to better track relationships between categorical features and then makes use of a standard MLP to output its final logits. In this paper we propose multiple modifications to the original TabTransformer performing better on binary classification tasks for three separate datasets with more than 1% AUROC gains. Inspired by gated MLP, linear projections are implemented in the MLP block and multiple acti…
Mixture-of-Experts (MoE), a conditional computation architecture, achieved promising performance by scaling local module (i.e. feed-forward network) of transformer. However, scaling the cross-token module (i.e. self-attention) is challenging due to the unstable training. This work proposes Sparse-MLP, an all-MLP model which applies sparsely-activated MLPs to cross-token modeling. Specifically, in each Sparse block of our all-MLP model, we apply two stages of MoE layers: one with MLP experts mixing information within channels along image patch dimension, the other with MLP experts mixing inform…
State University of New York at Buffalo（纽约州立大学布法罗分校）；Meta AI
All-MLP architectures have attracted increasing interest as an alternative to attention-based models. In NLP, recent work like gMLP shows that all-MLPs can match Transformers in language modeling, but still lag behind in downstream tasks. In this work, we analyze the limitations of MLPs in expressiveness, and propose sparsely activated MLPs with mixture-of-experts (MoEs) in both feature and input (token) dimensions. Such sparse all-MLPs significantly increase model capacity and expressiveness while keeping the compute constant. We address critical challenges in incorporating conditional comput…
University of Freiburg（弗莱堡大学）；Leibniz University Hannover（莱布尼茨汉诺威大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using …
Stanford University（斯坦福大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of California, Berkeley（加州大学伯克利分校）；École Polytechnique（巴黎高等理工学院）
Recent deep learning models for tabular data currently compete with the traditional ML models based on decision trees (GBDT). Unlike GBDT, deep models can additionally benefit from pretraining, which is a workhorse of DL for vision and NLP. For tabular problems, several pretraining methods were proposed, but it is not entirely clear if pretraining provides consistent noticeable improvements and what method should be used, since the methods are often not compared to each other or comparison is limited to the simplest MLP architectures. In this work, we aim to identify the best practices to pret…
M3-Net: A Cost-Effective Graph-Free MLP-Based Model for Traffic Prediction
National Innovative Institute of Defense Technology（国防技术创新国家研究院）；The Hong Kong University of Science（香港科学大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Peking University（北京大学）；Beijing Jiaotong University（北京交通大学）
Is the Statistical Advantage Worth the Cost? An Empirical Comparison of KANs and MLPs for Structured Data Classification
Institute of Computer Science, University of the Philippines Los Baños（菲律宾大学洛斯巴尼奥斯分校计算机科学研究所）；Machine Learning and Artificial Intelligence Applications Lab, University of the Philippines Los Baños（菲律宾大学洛斯巴尼奥斯分校机器学习与人工智能应用实验室）
Boston University（波士顿大学）；Sorbonne Université（索邦大学）；Ecole Polytechnique（巴黎综合理工学院）
Dedicated neural network (NN) architectures have been designed to handle specific data types (such as CNN for images or RNN for text), which ranks them among state-of-the-art methods for dealing with these data. Unfortunately, no architecture has been found for dealing with tabular data yet, for which tree ensemble methods (tree boosting, random forests) usually show the best predictive performances. In this work, we propose a new sparse initialization technique for (potentially deep) multilayer perceptrons (MLP): we first train a tree-based procedure to detect feature interactions and use the…
From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning
BASF（巴斯夫）；Clausthal University of Technology（克劳斯塔尔工业大学）；Amazon Music（亚马逊音乐）；Bielefeld School of Business, Hochschule Bielefeld (HSBI)（比勒费尔德商学院，比勒费尔德应用科学大学）
MoE-Enhanced Explainable Deep Manifold Transformation for Complex Data Embedding and Visualization
Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS and Westlake University（人工智能与机器人中心（CAIR），HKISI-CAS和西湖大学）；Westlake University（西湖大学）；School of Information and Electrical Engineering, Hangzhou City University（信息与电气工程学院，杭州市大学）；Academy of Edge Intelligence Hangzhou City University（边缘智能学院，杭州市大学）；CAIR, HKISI-CAS（人工智能与机器人中心（CAIR），HKISI-CAS）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA（多模态人工智能系统国家重点实验室（MAIS），CASIA）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（人工智能学院，中国科学院大学（UCAS））；Ant Group（蚂蚁集团）
Attention-Based Deep Learning for Early Parkinson's Disease Detection with Tabular Biomedical Data
African Institute for Mathematical Sciences, Kigali, Rwanda（非洲数学科学研究所，基加利，刚果（金））；Department of Data and Information Science, University of Ibadan, Oyo State, Nigeria（数据与信息科学系，伊巴丹大学，奥约州，尼日利亚）；Department of Computer Science and Information Systems, University of Limerick, Castletroy, V94 T9PX, Limerick, Ireland（计算机科学与信息系统系，利默里克大学，卡斯利特罗伊，爱尔兰）
N-BEATS-MOE: N-BEATS with a Mixture-of-Experts Layer for Heterogeneous Time Series Forecasting
Faculdade de Engenharia da Universidade do Porto, Porto, Portugal（葡萄牙波尔图大学工程学院）；Laboratory for Artificial Intelligence and Computer Science (LIACC)（人工智能与计算机科学实验室）
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China（计算机科学与技术学院，中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China, China（苏州先进研究院，中国科学技术大学）
School of Future Technology（未来技术学院）；Institute of Artificial Intelligence（人工智能研究院）；School of Computer Science and Technology（计算机科学与技术学院）；College of Sciences（科学学院）；SILC Business School（SILC商学院）
MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
School of Information Sciences, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校信息科学学院）；Hong Kong Baptist University（香港浸会大学）；Jacobs Technion-Cornell Institute, Cornell Tech（康奈尔理工学院雅各布斯以色列理工学院）
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，新加坡科技研究局）；Shanghai University of Engineering Science, China（上海工程技术大学）
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
Queen Mary University of London（伦敦玛丽女王大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；City University of London（伦敦城市大学）；National and Kapodistrian University of Athens（雅典国立卡波迪斯特里亚大学）；Archimedes AI, Athena RC（阿基米德人工智能公司，雅典娜研究中心）；The Cyprus Institute（塞浦路斯研究所）；Imperial College London（帝国理工学院）
