arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.00340cs.LGcs.CLcs.DC
收藏

解锁大语言模型训练中令牌过滤的全面效率

Unlocking Full Efficiency of Token Filtering in Large Language Model Training

Shanghai University of Finance and Economics(上海财经大学);Hong Kong University of Science and Technology(香港科技大学);University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学)

Di Chai, Pengbo Li, Feiyuan Zhang, Yilun Jin, Han Tian, Kaiqiang Xu, Binhang Yuan, Dian Shen, Junxue Zhang, Kai Chen

AI总结:

本文提出Centrifuge系统,通过算法与系统协同设计提升令牌过滤效率,减少反向传播时间达49.9%,端到端训练时间达34.7%,并提升模型性能26.6%。

摘要:

令牌过滤已被提出以通过训练过程中消除无用令牌来增强大语言模型(LLMs)的实用性。尽管使用更少的令牌预计会减少计算工作量,但现有方法尚未实现实际效率提升。这主要是由于两个因素:(1)现有工作在加速方面稀疏性不足,(2)令牌过滤在非标准的现有机器学习(ML)库中运行,因此无法高效支持。本文提出了Centrifuge,一个利用算法和系统协同设计的系统,以释放LLM训练中令牌过滤的全部效率。在算法层面,Centrifuge在注意力反向核中过滤无用令牌的激活,以在反向计算中增强稀疏性。在系统层面,Centrifuge提出一个自动工作流,将稀疏GEMM转换为维度减少的密集GEMM,以使用标准ML库优化效率。在1.1B到40B规模的模型上评估表明,当过滤50%的令牌时,Centrifuge将反向传播时间减少高达49.9%,端到端训练时间减少高达34.7%。实用性评估表明,Centrifuge保留了令牌过滤的实用性优势,并通过与标准训练相比提升模型性能高达26.6%。Centrifuge设计用于无缝集成到现有LLM训练框架中,使已经使用令牌过滤的系统通过只需一行代码即可加速训练。

英文摘要:

Token filtering has been proposed to enhance the utility of large language models (LLMs) by eliminating inconsequential tokens during training. While usingfewer tokens is expected to reduce computational workloads, existing methods have not yet achieved a real-world efficiency boost. This is primarily due to two factors: (1) existing work has inadequate sparsity for speedup, and (2) token filtering operates within a sparsity range that is non-standard in existing machine learning (ML) libraries and thus cannot be efficiently supported. This paper presents Centrifuge, a system that leverages algorithm and system co-design to unleash the full efficiency of token filtering in LLM training. At the algorithm level, Centrifuge filters activations of inconsequential tokens in the attention backward kernel to amplify the sparsity in backward computation. At the system level, Centrifuge proposes an automatic workflow that transforms sparse GEMM into dimension-reduced dense GEMM for optimized efficiency using standard ML libraries. Evaluations on models with various scales--from 1.1B to 40B--demonstrate that Centrifuge reduces backpropagation time by up to 49.9\% and end-to-end training time by up to 34.7\% when filtering 50\% of tokens. Utility assessments indicate that Centrifuge preserves the utility benefits of token filtering and significantly enhances model performance by up to 26.6\% compared to standard training. Centrifuge is designed for seamless integration into existing LLM training frameworks, enabling systems already utilizing token filtering to accelerate training with just one line of code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑