The increasing size of language models raises great research interests in parameter-efficient fine-tuning such as LoRA that freezes the pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks (e.g., summarization, question answering and translation). To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. The integrated framework is validated on two created deidentified medical report summarization datasets based on MIMIC-IV-Note and two public medical dialogue datasets. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase, while preserving over 92% generation qualities on free-text sequence-to-sequence tasks.
All-in-One Tuning and Structural Pruning for Domain-Specific LLMs
Northeastern University(东北大学);George Mason University(乔治梅森大学);GE HealthCare(GE医疗);University of Pennsylvania(宾夕法尼亚大学);University of Pittsburgh(匹兹堡大学);University of Florida(佛罗里达大学);Florida State University(佛罗里达州立大学)
PAT: Pruning-Aware Tuning for Large Language Models
School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);University of Arizona(亚利桑那大学);Samsung Electronic Research Centre of China(三星中国电子研究中心);Interdisciplinary Research Center for Future Intelligent Chips, Nanjing University(南京大学未来智能芯片交叉研究中心)
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm
University of Connecticut(康涅狄格大学);Penn State University(宾夕法尼亚州立大学);Moffett AI(莫菲特人工智能公司);Northeastern University(东北大学);Stevens Institute of Technology(史蒂文斯理工学院);University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校)
Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏州高等研究院)
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy(中关村学院);University of Science and Technology of China(中国科学技术大学)
Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
Southeast University(东南大学);Peking University(北京大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Technische Universität Chemnitz(开姆尼茨工业大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models
Korea Advanced Institute of Science and Technology(韩国科学技术院);University of Tokyo, Tokyo, Japan(东京大学);Tokyo Institute of Technology, Tokyo, Japan(东京技术大学)
Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs
School of Electronics and Computer Science, University of Southampton, UK(索姆塞特大学电子与计算机科学学院);Department of Computer Science, University of Liverpool, UK(利弗pool大学计算机科学系)
MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);City University of Hong Kong(香港城市大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学)
From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression
Alibaba(阿里巴巴);Peking University(北京大学)
预训练语言模型(Pre-trained Language Models,PLM)在预训练和微调的范式下在各种自然语言处理(Natural Language Processing,NLP)任务中取得了巨大的成功。PLM具有大量的参数,计算密集型和资源消耗型。因此,模型修剪已被引入到压缩大规模PLM。然而,大多数已有的方法在剪枝时只考虑任务相关的知识,而忽略了任务无关的基本知识,这可能会导致灾难性遗忘问题,从而导致泛化能力差。为了在我们的修剪模型中保持任务不可知和任务特定的知识,我们提出了预训练和微调范式下的主动修剪(CAP)。它被设计为一个通用框架,兼容结构化和非结构化修剪。CAP与对比学习相结合,使修剪后的模型能够从预训练的模型中学习任务不可知的知识,并对任务特定的知识进行微调。此外,为了更好地保持修剪模型的性能,快照(即,在每次修剪迭代的中间模型)也用作修剪的有效监督。我们广泛的实验表明,采用CAP一贯产生显着的改善,特别是在极高的稀疏性的情况下。仅保留3%的模型参数(即,97%稀疏性),CAP成功地实现了99.2%和96.3%的原始BERT性能在QQP和MNLI任务。此外,我们的探测实验表明,CAP修剪的模型倾向于实现更好的泛化能力。
LEAP: Learnable Pruning for Transformer-based Models
Microsoft(微软);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Berkeley(加州大学伯克利分校);International Computer Science Institute(国际计算机科学研究所)
O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning
China Agriculture University(中国农业大学);Tsinghua University(清华大学);University of Oxford(牛津大学);Didichuxing Co. Ltd(滴滴出行有限公司);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Nanyang Technological University(南洋理工大学)