arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.06571cs.CLcs.AI
收藏

SUBLLM:一种面向LLM的基于词元序列下采样的新型高效架构

SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM

Xiaomi AI Lab(小米AI实验室);School of Informatics, Xiamen University(厦门大学信息学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)

Quandong Wang, Yuxuan Yuan, Xiaoyu Yang, Ruike Zhang, Kang Zhao, Wei Liu, Jian Luan, Daniel Povey, Bin Wang

AI总结:

针对LLM训练与推理效率的挑战,提出SUBLLM架构,通过在纯解码器框架中引入下采样、上采样和旁路模块,在保持少样本性能的同时显著提升了训练与推理速度并降低了内存消耗。

摘要:

尽管大型语言模型(LLM)在各个领域取得了显著成功,但其训练和推理的效率仍然是一个主要挑战。为了解决这个问题,我们提出了SUBLLM,即下采样-上采样-旁路大型语言模型(Subsampling-Upsampling-Bypass Large Language Model)的缩写。这是一种创新的架构,通过结合下采样、上采样和旁路模块来扩展核心的纯解码器框架。下采样模块负责缩短序列,上采样模块恢复序列长度,旁路模块则增强收敛性。与LLaMA相比,所提出的SUBLLM在训练和推理速度以及内存使用方面均展现出显著提升,同时保持了具有竞争力的少样本性能。在训练期间,SUBLLM将速度提高了26%,并使每块GPU的内存减少10GB。在推理中,它将速度提升高达37%,并将每块GPU的内存减少1GB。当上下文窗口扩展至8192时,其训练和推理速度可分别提升34%和52%。我们的代码已在 https://github.com/XiaoMi/subllm 提供。

英文摘要:

While Large Language Models (LLMs) have achieved remarkable success in various fields, the efficiency of training and inference remains a major challenge. To address this issue, we propose SUBLLM, short for Subsampling-Upsampling-Bypass Large Language Model, an innovative architecture that extends the core decoder-only framework by incorporating subsampling, upsampling, and bypass modules. The subsampling modules are responsible for shortening the sequence, while the upsampling modules restore the sequence length, and the bypass modules enhance convergence. In comparison to LLaMA, the proposed SUBLLM exhibits significant enhancements in both training and inference speeds as well as memory usage, while maintaining competitive few-shot performance. During training, SUBLLM increases speeds by 26% and cuts memory by 10GB per GPU. In inference, it boosts speeds by up to 37% and reduces memory by 1GB per GPU. The training and inference speeds can be enhanced by 34% and 52% respectively when the context window is expanded to 8192. Our code is available at https://github.com/XiaoMi/subllm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑