Mixture-of-Experts (MoEs) can scale up beyond traditional deep learning models by employing a routing strategy in which each input is processed by a single "expert" deep learning model. This strategy allows us to scale up the number of parameters defining the MoE while maintaining sparse activation, i.e., MoEs only load a small number of their total parameters into GPU VRAM for the forward pass depending on the input. In this paper, we provide an approximation and learning-theoretic analysis of mixtures of expert MLPs with (P)ReLU activation functions. We first prove that for every error level $\varepsilon>0$ and every Lipschitz function $f:[0,1]^n\to \mathbb{R}$, one can construct a MoMLP model (a Mixture-of-Experts comprising of (P)ReLU MLPs) which uniformly approximates $f$ to $\varepsilon$ accuracy over $[0,1]^n$, while only requiring networks of $\mathcal{O}(\varepsilon^{-1})$ parameters to be loaded in memory. Additionally, we show that MoMLPs can generalize since the entire MoMLP model has a (finite) VC dimension of $\tilde{O}(L\max\{nL,JW\})$, if there are $L$ experts and each expert has a depth and width of $J$ and $W$, respectively.
Looped ReLU MLPs May Be All You Need as Practical Programmable Computers
The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Tsinghua University（清华大学）；The Simons Institute for the Theory of Computing（西蒙斯理论计算研究所）；University of Pennsylvania（宾夕法尼亚大学）
Sharif University of Technology（谢里夫理工大学）；Electronics Research Institute, Sharif University of Technology（谢里夫理工大学电子研究所）；McGill University（麦吉尔大学）
In this paper, a novel multi-head multi-layer perceptron (MLP) structure is presented for implicit neural representation (INR). Since conventional rectified linear unit (ReLU) networks are shown to exhibit spectral bias towards learning low-frequency features of the signal, we aim at mitigating this defect by taking advantage of the local structure of the signals. To be more specific, an MLP is used to capture the global features of the underlying generator function of the desired signal. Then, several heads are utilized to reconstruct disjoint local features of the signal, and to reduce the c…
Department of Computer Science Rensselaer Polytechnic Institute（计算机科学系伦塞拉尔理工大学）；Department of Biomedical Engineering Rensselaer Polytechnic Institute（生物医学工程系伦塞拉尔理工大学）
MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
School of Information Sciences, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校信息科学学院）；Hong Kong Baptist University（香港浸会大学）；Jacobs Technion-Cornell Institute, Cornell Tech（康奈尔理工学院雅各布斯以色列理工学院）
An appropriate choice of the activation function (like ReLU, sigmoid or swish) plays an important role in the performance of (deep) multilayer perceptrons (MLP) for classification and regression learning. Prototype-based classification learning methods like (generalized) learning vector quantization (GLVQ) are powerful alternatives. These models also deal with activation functions but here they are applied to the so-called classifier function instead. In this paper we investigate successful candidates of activation functions known for MLPs for application in GLVQ and their influence on the per…
Tel Aviv University（特拉维夫大学）；New York University（纽约大学）
A major factor in the success of deep neural networks is the use of sophisticated architectures rather than the classical multilayer perceptron (MLP). Residual networks (ResNets) stand out among these powerful modern architectures. Previous works focused on the optimization advantages of deep ResNets over deep MLPs. In this paper, we show another distinction between the two models, namely, a tendency of ResNets to promote smoother interpolations than MLPs. We analyze this phenomenon via the neural tangent kernel (NTK) approach. First, we compute the NTK for a considered ResNet model and prove …
Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference
Chongqing Key Laboratory of Secure Computing for Biology（重庆生物安全计算重点实验室）；Chongqing Institute of Green and Intelligent Technology（重庆绿色智能技术研究所）；Chinese Academy of Sciences（中国科学院）；Department of Industrial and Systems Engineering（工业与系统工程系）
Rethinking Nonlinearity: Trainable Gaussian Mixture Modules for Modern Neural Architectures
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Great Bay University（大湾大学）；University of Massachusetts at Amherst（马萨诸塞大学阿姆赫斯特分校）
Mixture-of-Experts (MoE), a conditional computation architecture, achieved promising performance by scaling local module (i.e. feed-forward network) of transformer. However, scaling the cross-token module (i.e. self-attention) is challenging due to the unstable training. This work proposes Sparse-MLP, an all-MLP model which applies sparsely-activated MLPs to cross-token modeling. Specifically, in each Sparse block of our all-MLP model, we apply two stages of MoE layers: one with MLP experts mixing information within channels along image patch dimension, the other with MLP experts mixing inform…
Mixture of experts method is a neural network based ensemble learning that has great ability to improve the overall classification accuracy. This method is based on the divide and conquer principle, in which the problem space is divided between several experts by supervisition of gating network. In this paper, we propose an ensemble learning method based on mixture of experts which is named mixture of ELM based experts with trainable gating network (MEETG) to improve the computing cost and to speed up the learning process of ME. The structure of ME consists of multi layer perceptrons (MLPs) as…
University of Southern California（南加州大学）；Army Research Laboratory（陆军研究实验室）
A closed-form solution exists in two-class linear discriminant analysis (LDA), which discriminates two Gaussian-distributed classes in a multi-dimensional feature space. In this work, we interpret the multilayer perceptron (MLP) as a generalization of a two-class LDA system so that it can handle an input composed by multiple Gaussian modalities belonging to multiple classes. Besides input layer $l_{in}$ and output layer $l_{out}$, the MLP of interest consists of two intermediate layers, $l_1$ and $l_2$. We propose a feedforward design that has three stages: 1) from $l_{in}$ to $l_1$: half-spac…
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China（数学科学国家重点实验室，数学与系统科学研究院，中国科学院，北京100190，中国）；School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China（中国科学院大学数学科学学院，北京100049，中国）；Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China（上海数学与交叉科学研究所（SIMIS），上海，中国）；Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China（浙江省系统健康科学重点实验室，生命科学学院，杭州先进研究院，中国科学院大学，中国科学院，杭州310024，中国）
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，新加坡科技研究局）；Shanghai University of Engineering Science, China（上海工程技术大学）
Accurate demand forecasts can help on-line retail organizations better plan their supply-chain processes. The challenge, however, is the large number of associative factors that result in large, non-stationary shifts in demand, which traditional time series and regression approaches fail to model. In this paper, we propose a Neural Network architecture called AR-MDN, that simultaneously models associative factors, time-series trends and the variance in the demand. We first identify several causal features and use a combination of feature embeddings, MLP and LSTM to represent them. We then mode…
Quantitative Approximation Rates for Group Equivariant Learning
Department of Mathematics（数学系）；Texas A&M University（德克萨斯A&M大学）；Faculty of Mathematics（数学系）；Technion- Israel Institute of Technology（技术学院-以色列理工学院）；Computer Science and Artificial Intelligence Laboratory（计算机科学与人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）
M3-Net: A Cost-Effective Graph-Free MLP-Based Model for Traffic Prediction
National Innovative Institute of Defense Technology（国防技术创新国家研究院）；The Hong Kong University of Science（香港科学大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Peking University（北京大学）；Beijing Jiaotong University（北京交通大学）
Towards the Connection between Activation Sparsity and Flat Minima
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Institute of Brain-Machine Interface, Nanjing University（南京大学脑机接口研究院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
