The considerable size of Large Language Models (LLMs) presents notable deployment challenges, particularly on resource-constrained hardware. Structured pruning, offers an effective means to compress LLMs, thereby reducing storage costs and enhancing inference speed for more efficient utilization. In this work, we study data-efficient and resource-efficient structure pruning methods to obtain smaller yet still powerful models. Knowledge Distillation is well-suited for pruning, as the intact model can serve as an excellent teacher for pruned students. However, it becomes challenging in the context of LLMs due to memory constraints. To address this, we propose an efficient progressive Numerous-teacher pruning method (NutePrune). NutePrune mitigates excessive memory costs by loading only one intact model and integrating it with various masks and LoRA modules, enabling it to seamlessly switch between teacher and student roles. This approach allows us to leverage numerous teachers with varying capacities to progressively guide the pruned model, enhancing overall performance. Extensive experiments across various tasks demonstrate the effectiveness of NutePrune. In LLaMA-7B zero-shot experiments, NutePrune retains 97.17% of the performance of the original model at 20% sparsity and 95.07% at 25% sparsity. Our code is available at https://github.com/Lucius-lsr/NutePrune.
Beyond Solo and Consistency: Vindicating Multi-Agent Debate via Conditional Progressive Pruning
Rutgers University, New Brunswick（罗格斯大学新布朗斯维克分校）；University of Cambridge（剑桥大学）；Tsinghua University（清华大学）；Harvard University（哈佛大学）；Case Western Reserve University（凯斯西储大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）；Carnegie Mellon University（卡内基梅隆大学）
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm
University of Connecticut（康涅狄格大学）；Penn State University（宾夕法尼亚州立大学）；Moffett AI（莫菲特人工智能公司）；Northeastern University（东北大学）；Stevens Institute of Technology（史蒂文斯理工学院）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoning
Institute of Cyberspace Security, Zhejiang University of Technology A STAR Amazon Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology
Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
School of Computer Science and Engineering, Beihang University, Beijing, China（计算机科学与工程学院，北航，北京，中国）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China（虚拟现实技术与系统国家重点实验室，北航，北京，中国）
To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation
College of AI, Tsinghua University（清华大学人工智能学院）；Proxseer Inc.（Proxseer公司）；School of Computer Science, Nanjing University（南京大学计算机科学与技术学院）；Software Institute, Nanjing University（南京大学软件学院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
