自适应深度稀疏框架：基于相似性驱动的预训练语言模型资源分配
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
浏览论文内容
中文总结 AI 辅助
研究针对预训练语言模型推理成本高的问题，提出自适应深度稀疏框架AdaDSF，基于层输入输出隐藏状态余弦相似性分配令牌保留率，用轻量级路由器选择信息令牌，在多任务中大幅降推理FLOPs且精度退化小。
中文摘要 AI 辅助
大型语言模型（LLMs）虽性能强大，但Transformer架构推理成本高。现有加速方法依赖特定任务微调或从头训练，增加适配成本并限制跨任务可用性。我们提出自适应深度稀疏框架（AdaDSF），无需完全重新训练就能将现成预训练LLMs转换为深度稀疏模型。关键在于各层对表示转换贡献不均，通过层输入与输出隐藏状态的余弦相似性表征。基于此，AdaDSF从相似性统计中分配逐层令牌保留率，用轻量级路由器选择信息令牌，引入特征保留对齐目标匹配稀疏与密集模型的中间和最终表示。在语言建模和常识推理任务中，AdaDSF大幅降低推理FLOPs且性能接近密集模型，在可比稀疏性下，其精度退化比MoD、D-LLM和DLO等基线更小。
英文摘要
Large language models (LLMs) achieve strong generation and reasoning performance, but the Transformer architecture incurs high inference cost. Existing acceleration methods often rely on task-specific fine-tuning or training from scratch, increasing adaptation cost and limiting cross-task usability. We present an Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained LLMs into depth-sparse models without full retraining. Our key insight is that layers contribute unequally to representation transformation, characterized by the cosine similarity between layer input and output hidden states. Based on this, AdaDSF assigns layer-wise token retention ratios from similarity statistics, uses a lightweight router to select informative tokens at each layer, and introduces a feature-preserving alignment objective to match intermediate and final representations between sparse and dense models. On GPT-NeoX and Qwen2.5 over language modeling and commonsense reasoning, AdaDSF substantially reduces inference FLOPs while preserving performance close to dense counterparts. Under comparable sparsity, AdaDSF consistently yields smaller accuracy degradation than strong baselines including MoD, D-LLM, and DLO.
发表机构
- Southern University of Science and Technology（南方科技大学）
- Huawei Technologies Co., Ltd.（华为技术有限公司）
机构由 AI 辅助整理，请以论文原文为准。
