Large neural networks excel in many domains, but they are expensive to train and fine-tune. A popular approach to reduce their compute or memory requirements is to replace dense weight matrices with structured ones (e.g., sparse, low-rank, Fourier transform). These methods have not seen widespread adoption (1) in end-to-end training due to unfavorable efficiency--quality tradeoffs, and (2) in dense-to-sparse fine-tuning due to lack of tractable algorithms to approximate a given dense weight matrix. To address these issues, we propose a class of matrices (Monarch) that is hardware-efficient (they are parameterized as products of two block-diagonal matrices for better hardware utilization) and expressive (they can represent many commonly used transforms). Surprisingly, the problem of approximating a dense weight matrix with a Monarch matrix, though nonconvex, has an analytical optimal solution. These properties of Monarch matrices unlock new ways to train and fine-tune sparse and dense models. We empirically validate that Monarch can achieve favorable accuracy-efficiency tradeoffs in several end-to-end sparse training applications: speeding up ViT and GPT-2 training on ImageNet classification and Wikitext-103 language modeling by 2x with comparable model quality, and reducing the error on PDE solving and MRI reconstruction tasks by 40%. In sparse-to-dense training, with a simple technique called "reverse sparsification," Monarch matrices serve as a useful intermediate representation to speed up GPT-2 pretraining on OpenWebText by 2x without quality drop. The same technique brings 23% faster BERT pretraining than even the very optimized implementation from Nvidia that set the MLPerf 1.1 record. In dense-to-sparse fine-tuning, as a proof-of-concept, our Monarch approximation algorithm speeds up BERT fine-tuning on GLUE by 1.7x with comparable accuracy.
Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);ShanghaiTech University(上海科技大学)
Scaling Probabilistic Circuits via Monarch Matrices
Department of Computer Science, University of California, Los Angeles(加州大学洛杉矶分校计算机科学系);Department of Computer Science, Stanford University(斯坦福大学计算机科学系)
SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks
Sapienza University of Rome(罗马大学);Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt(达姆施塔特工业大学计算机科学系泛在知识处理实验室);National Research Center for Applied Cybersecurity ATHENE(国家应用网络安全研究中心ATHENE);Technical University of Denmark(丹麦技术大学);TU Darmstadt & Hessian Center for AI (hessian.AI)(达姆施塔特工业大学与黑森人工智能中心);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);University of Central Florida(中佛罗里达大学)
Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
Georgia Institute of Technology(佐治亚理工学院);National Taiwan University(国立台湾大学);University of Palermo(帕尔米奥大学);NVIDIA Research(NVIDIA研究);Hon-Hai Quantum Computing Research Center(鸿海量子计算研究中心)