Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation
School of Statistics and Data Science, Nankai University(南开大学统计与数据科学学院);School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Department of Automation, Tsinghua University(清华大学自动化系);Central Research Institute, United Imaging Healthcare, Co., Ltd.(联影医疗股份有限公司中央研究院)
Fine-tuning pretrained large models to downstream tasks is an important problem, which however suffers from huge memory overhead due to large-scale parameters. This work strives to reduce memory overhead in fine-tuning from perspectives of activation function and layer normalization. To this end, we propose the Approximate Backpropagation (Approx-BP) theory, which provides the theoretical feasibility of decoupling the forward and backward passes. We apply our Approx-BP theory to backpropagation training and derive memory-efficient alternatives of GELU and SiLU activation functions, which use derivative functions of ReLUs in the backward pass while keeping their forward pass unchanged. In addition, we introduce a Memory-Sharing Backpropagation strategy, which enables the activation memory to be shared by two adjacent layers, thereby removing activation memory usage redundancy. Our method neither induces extra computation nor reduces training efficiency. We conduct extensive experiments with pretrained vision and language models, and the results demonstrate that our proposal can reduce up to $\sim$$30\%$ of the peak memory usage. Our code is released at https://github.com/yyyyychen/LowMemoryBP.
School of Computing, Montclair State University(蒙特克莱尔州立大学计算机学院);Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学计算机科学与工程系)
Randomized Matrix Sketching for Neural Network Training and Gradient Monitoring
Department of Mathematical Sciences(数学科学系);Center for Mathematics(数学中心);Artificial Intelligence (CMAI), George Mason University. Fairfax, VA 22030.(人工智能(CMAI),乔治·马歇尔大学,弗吉尼亚州 Fairfax,22030);School of Mathematical(数学系);Statistical Sciences, Clemson University, Clemson, SC 29634.(统计科学系,克莱姆森大学,SC州 Clemson,29634)
DictPFL: Efficient and Private Federated Learning on Encrypted Gradients
University of Central Florida(佛罗里达中央大学);Florida State University(佛罗里达州立大学);Old Dominion University(旧 Dominion 大学);University of Texas Health Science Center at Houston(德克萨斯大学健康科学中心休斯顿分校)