arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.00595cs.LG
收藏

Monarch:高效准确的表达结构矩阵

Monarch: Expressive Structured Matrices for Efficient and Accurate Training

Stanford University(斯坦福大学);University at Buffalo, SUNY(纽约州立大学布法罗分校);University of Michigan(密歇根大学)

Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, Christopher Ré

摘要:

大型神经网络在许多领域都很出色,但它们的训练和微调成本很高。一种减少其计算或存储器需求的流行方法是用结构化的权重矩阵(例如,稀疏、低秩、傅立叶变换)。这些方法尚未被广泛采用:(1)由于不利的效率-质量权衡,在端到端训练中,以及(2)由于缺乏易于处理的算法来近似给定的密集权重矩阵,在密集到稀疏微调中。为了解决这些问题,我们提出了一类矩阵(君主),这是硬件高效(它们被参数化为两个块对角矩阵的产品,以更好的硬件利用率)和表达(他们可以代表许多常用的变换)。令人惊讶的是,用Monarch矩阵近似稠密权重矩阵的问题,虽然是非凸的,但有一个分析最优解。Monarch矩阵的这些属性为训练和微调稀疏和密集模型提供了新的方法。我们通过经验验证,Monarch可以在几个端到端稀疏训练应用中实现良好的准确性-效率权衡:将ImageNet分类和Wikitext-103语言建模的ViT和GPT-2训练速度提高2倍,模型质量相当,并将PDE求解和MRI重建任务的错误减少40%。在稀疏到密集的训练中,使用一种称为“反向稀疏化”的简单技术,Monarch矩阵作为一种有用的中间表示,可以在不降低质量的情况下将OpenWebText上的GPT-2预训练速度提高2倍。同样的技术带来了23%的速度比BERT预训练,甚至比Nvidia非常优化的实现,设置MLPerf 1.1记录。在从密集到稀疏的微调中,作为概念验证,我们的Monarch近似算法将GLUE的BERT微调速度提高了1.7倍,并且具有相当的精度。

英文摘要:

Large neural networks excel in many domains, but they are expensive to train and fine-tune. A popular approach to reduce their compute or memory requirements is to replace dense weight matrices with structured ones (e.g., sparse, low-rank, Fourier transform). These methods have not seen widespread adoption (1) in end-to-end training due to unfavorable efficiency--quality tradeoffs, and (2) in dense-to-sparse fine-tuning due to lack of tractable algorithms to approximate a given dense weight matrix. To address these issues, we propose a class of matrices (Monarch) that is hardware-efficient (they are parameterized as products of two block-diagonal matrices for better hardware utilization) and expressive (they can represent many commonly used transforms). Surprisingly, the problem of approximating a dense weight matrix with a Monarch matrix, though nonconvex, has an analytical optimal solution. These properties of Monarch matrices unlock new ways to train and fine-tune sparse and dense models. We empirically validate that Monarch can achieve favorable accuracy-efficiency tradeoffs in several end-to-end sparse training applications: speeding up ViT and GPT-2 training on ImageNet classification and Wikitext-103 language modeling by 2x with comparable model quality, and reducing the error on PDE solving and MRI reconstruction tasks by 40%. In sparse-to-dense training, with a simple technique called "reverse sparsification," Monarch matrices serve as a useful intermediate representation to speed up GPT-2 pretraining on OpenWebText by 2x without quality drop. The same technique brings 23% faster BERT pretraining than even the very optimized implementation from Nvidia that set the MLPerf 1.1 record. In dense-to-sparse fine-tuning, as a proof-of-concept, our Monarch approximation algorithm speeds up BERT fine-tuning on GLUE by 1.7x with comparable accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑