Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Ezhou Industrial Technology Research Institute, Huazhong University of Science and Technology（华中科技大学鄂州工业技术研究院）；Department of CSE, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；China-Austria Belt and Road Joint Laboratory on AI and AM, Hangzhou Dianzi University（杭州电子科技大学中奥人工智能与增材制造一带一路联合实验室）；Center for Applied Statistics and School of Statistics, Renmin University of China（中国人民大学应用统计中心与统计学院）
Zenan Ling, Longbo Li, Zhanbo Feng, Yixuan Zhang, Feng Zhou, Robert C. Qiu, Zhenyu Liao
Deep equilibrium models (DEQs), as a typical implicit neural network, have demonstrated remarkable success on various tasks. There is, however, a lack of theoretical understanding of the connections and differences between implicit DEQs and explicit neural network models. In this paper, leveraging recent advances in random matrix theory (RMT), we perform an in-depth analysis on the eigenspectra of the conjugate kernel (CK) and neural tangent kernel (NTK) matrices for implicit DEQs, when the input data are drawn from a high-dimensional Gaussian mixture. We prove, in this setting, that the spectral behavior of these Implicit-CKs and NTKs depend on the DEQ activation function and initial weight variances, but only via a system of four nonlinear equations. As a direct consequence of this theoretical result, we demonstrate that a shallow explicit network can be carefully designed to produce the same CK or NTK as a given DEQ. Despite derived here for Gaussian mixture data, empirical results show the proposed theory and design principle also apply to popular real-world datasets.
Mixture of experts models for multilevel data: modelling framework and approximation theory
Georgia State University（佐治亚州立大学）；University of Toronto（多伦多大学）
多级数据在许多实际应用中很普遍。然而，它仍然是一个开放的研究问题，以确定和证明一类模型，灵活地捕捉范围广泛的多级数据。基于混合专家模型（mixed of experts，MoE）在回归数据拟合中的通用性，本文对MoE进行了扩展，研究了一类多水平数据的混合MoE（mixed MoE）模型.在一定的正则性条件下，证明了MMoE在任意连续混合效应模型空间中的弱收敛稠密性。因此，MMoE有可能准确地类似于几乎所有的特征继承在多级数据，包括边缘分布，相关结构，回归链接，随机截距和随机斜率。在一个特定的情况下，多层次的数据是分层的，我们进一步表明，嵌套版本的MMoE普遍近似于广泛的依赖结构的随机效应之间的不同因素水平。
Robust Federated Personalised Mean Estimation for the Gaussian Mixture Model
School of Technology \& Comp. Sc. Tata Institute of Fundamental Research Mumbai, India；Department of Electrical Engineering University of California Los Angeles, USA
