Large Language Models (LLMs) are difficult to fully fine-tune (e.g., with instructions or human feedback) due to their sheer number of parameters. A family of parameter-efficient sparse fine-tuning methods have proven promising in terms of performance but their memory requirements increase proportionally to the size of the LLMs. In this work, we scale sparse fine-tuning to state-of-the-art LLMs like LLaMA 2 7B and 13B. We propose SpIEL, a novel sparse fine-tuning method which, for a desired density level, maintains an array of parameter indices and the deltas of these parameters relative to their pretrained values. It iterates over: (a) updating the active deltas, (b) pruning indices (based on the change of magnitude of their deltas) and (c) regrowth of indices. For regrowth, we explore two criteria based on either the accumulated gradients of a few candidate parameters or their approximate momenta estimated using the efficient SM3 optimizer. We experiment with instruction-tuning of LLMs on standard dataset mixtures, finding that SpIEL is often superior to popular parameter-efficient fine-tuning methods like LoRA (low-rank adaptation) in terms of performance and comparable in terms of run time. We additionally show that SpIEL is compatible with both quantization and efficient optimizers, to facilitate scaling to ever-larger model sizes. We release the code for SpIEL at https://github.com/AlanAnsell/peft and for the instruction-tuning experiments at https://github.com/ducdauge/sft-llm.
Sparse is Enough in Fine-tuning Pre-trained Large Language Models
National Engineering Research Center for Multimedia Software(国家多媒体软件工程技术研究中心);School of Computer Science, Wuhan University(武汉大学计算机学院);Hubei Luojia Laboratory(湖北珞珈实验室);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系)
Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution
Eindhoven University of Technology(埃因霍温理工大学);University of Cambridge(剑桥大学);University of Luxembourg(卢森堡大学);University of Twente(埃因霍温理工大学);University of Surrey(萨里大学);Tübingen AI Center(图宾根人工智能中心);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所)
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China.(计算机科学与工程系,南方科技大学,深圳,中国);Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.(数学系,南方科技大学,深圳,中国)
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS(中国科学院计算技术研究所智能信息处理重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Peng Cheng Laboratory(鹏城实验室)
GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection
Ant Group(蚂蚁集团);Cornell University(康奈尔大学);University of Liverpool(利物浦大学);Soochow University(苏州大学);Cleveland Clinic Lerner Research Institution(克利夫兰诊所勒纳研究机构)
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Department of Linguistics and Translation, City University of Hong Kong(香港城市大学语言学及翻译学系)
Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models
Southern University of Science and Technology(南方科技大学);Shenzhen Institutes of Advanced Technology, CAS(深圳先进技术研究院);University of Birmingham(伯明翰大学);Imperial College London(伦敦帝国理工学院);University of Edinburgh(爱丁堡大学);Shenzhen University of Advanced Technology(深圳大学)
Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs
Huawei Technologies(华为技术有限公司);University of Oxford(牛津大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Eindhoven University of Technology(埃因霍温理工大学);Xiamen University(厦门大学);Tencent Youtu Lab(腾讯优图实验室);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能研究所)
Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts
City University of Hong Kong(香港城市大学);Korea Advanced Institute of Science and Technology(韩国科学技术院);University of Oxford(牛津大学);Harvard University(哈佛大学);Nanyang Technological University(南洋理工大学)
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
Tennessee Tech University, \ Alamos National Laboratory Cookeville Tennessee USA;Los Alamos National Laboratory Los Alamos New Mexico USA;Tennessee Tech University Cookeville Tennessee USA;Tennessee Tech University, \ Alamos National Laboratory;Los Alamos National Laboratory;Tennessee Tech University
Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
Beijing Laboratory of Advanced Information Network(北京先进信息网络实验室);Beijing Key Laboratory of Network System Architecture and Convergence(北京网络系统架构与融合重点实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Department of Electrical and Computer Engineering and Institute for Data Science and Computing(电气与计算机工程系和数据科学与计算研究所);University of Miami(迈阿密大学);Information Systems Technology and Design Pillar(信息系统技术与设计支柱)
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search
TJUNLP Lab, College of Intelligence and Computing, Tianjin University(天津大学智能计算学院);School of New Media and Communication, Tianjin University(天津大学新媒体与传播学院)