Existing low-rank adaptation (LoRA) methods face challenges on sparse large language models (LLMs) due to the inability to maintain sparsity. Recent works introduced methods that maintain sparsity by augmenting LoRA techniques with additional masking mechanisms. Despite these successes, such approaches suffer from an increased memory and computation overhead, which affects efficiency of LoRA methods. In response to this limitation, we introduce LoRS, an innovative method designed to achieve both memory and computation efficiency when fine-tuning sparse LLMs. To mitigate the substantial memory and computation demands associated with preserving sparsity, our approach incorporates strategies of weight recompute and computational graph rearrangement. In addition, we also improve the effectiveness of LoRS through better adapter initialization. These innovations lead to a notable reduction in memory and computation consumption during the fine-tuning phase, all while achieving performance levels that outperform existing LoRA approaches.
Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models
Saarland University(萨尔兰大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心)
Bayesian Low-Rank Factorization for Robust Model Adaptation
Interactive Systems Lab, Karlsruhe Institut of Technology (KIT), Germany(交互系统实验室,卡尔斯鲁厄理工学院(KIT),德国);InterACT, Carnegie Mellon University (CMU), USA(InterACT,卡内基梅隆大学(CMU),美国)
Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models
Southern University of Science and Technology(南方科技大学);Shenzhen Institutes of Advanced Technology, CAS(深圳先进技术研究院);University of Birmingham(伯明翰大学);Imperial College London(伦敦帝国理工学院);University of Edinburgh(爱丁堡大学);Shenzhen University of Advanced Technology(深圳大学)