LLMs demonstrate significant inference capacities in complicated machine learning tasks, using the Transformer model as its backbone. Motivated by the limited understanding of such models on the unsupervised learning problems, we study the learning guarantees of Transformers in performing multi-class clustering of the Gaussian Mixture Models. We develop a theory drawing strong connections between the Softmax Attention layers and the workflow of the EM algorithm on clustering the mixture of Gaussians. Our theory provides approximation bounds for the Expectation and Maximization steps by proving the universal approximation abilities of multivariate mappings by Softmax functions. In addition to the approximation guarantees, we also show that with a sufficient number of pre-training samples and an initialization, Transformers can achieve the minimax optimal rate for the problem considered. Our extensive simulations empirically verified our theory by revealing the strong learning capacities of Transformers even beyond the assumptions in the theory, shedding light on the powerful inference capacities of LLMs.
Sorbonne Université, LPSM Paris, France（索邦大学，巴黎LPSM分校，法国）；EPFL, Institute of Mathematics Lausanne, Switzerland（苏黎世联邦理工学院，数学研究所，瑞士）；Laboratoire de Mathématiques d’Orsay Université Paris Saclay（奥赛数学实验室，巴黎萨克雷大学）；Institut Universitaire de France（法国国家科学研究中心）
Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers
Carnegie Mellon University, Pittsburgh, USA；University of Southern California, Los Angeles, CA, USA；University of California, San Diego, Department of Mathematics, La Jolla, CA, USA；University of California, Davis, Applied Mathematics \& Statistics, Davis, CA, USA；Carnegie Mellon University, Pittsburgh, PA, USA
Bipartite Graph Attention-based Clustering for Large-scale scRNA-seq Data
Institute of Intelligent Information Processing, Shanxi University（山西大学智能信息处理研究院）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学阿联酋商业学院）
Mass Distribution versus Density Distribution in the Context of Clustering
State Key Laboratory for Novel Software Technology, Nanjing University, China 210023（新型软件技术国家重点实验室，南京大学，中国）；School of Artificial Intelligence, Nanjing University, Nanjing, China 210023（人工智能学院，南京大学，南京，中国）；School of IT, Deakin University, Geelong, Australia 3125（信息技术学院，德金大学，Geelong，澳大利亚）
Quantitative Clustering in Mean-Field Transformer Models
Department of Mathematics, Massachusetts Institute of Technology（数学系，麻省理工学院）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（电气工程与计算机科学系，麻省理工学院）
