arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.00518cs.LGcs.AIcs.CL
收藏

EE-Tuning:一种用于调优早退大语言模型的经济且可扩展的解决方案

EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models

Alibaba Group(阿里巴巴集团)

Xuchen Pan, Yanxi Chen, Yaliang Li, Bolin Ding, Jingren Zhou

AI总结:

本研究提出轻量经济型的EE-Tuning方案,以参数高效方式为预训练LLM增设早退层并调优,兼容3D并行,在有限预算下实现有效早退推理,且已开源代码。

摘要:

本研究提出EE-Tuning,这是一种用于训练/调优早退大语言模型(LLM)的轻量经济型方案。与常见的全参数预训练方法不同,EE-Tuning为任意预训练(且可能经过微调)的标准LLM增设额外的早退层,以参数高效的方式进行调优,所需计算资源与训练数据显著更少。我们实现的EE-Tuning通过大量性能优化实现了出色的训练效率,且因完全兼容3D并行而具备可扩展性。系统实验结果验证了EE-Tuning的有效性,证实仅需有限训练预算即可实现有效的早退LLM推理。为让社区能够使用早退LLM,我们在https://github.com/pan-x-c/EE-LLM 发布了EE-Tuning实现的源代码。

英文摘要:

This work introduces EE-Tuning, a lightweight and economical solution to training/tuning early-exit large language models (LLMs). In contrast to the common approach of full-parameter pre-training, EE-Tuning augments any pre-trained (and possibly fine-tuned) standard LLM with additional early-exit layers that are tuned in a parameter-efficient manner, which requires significantly less computational resources and training data. Our implementation of EE-Tuning achieves outstanding training efficiency via extensive performance optimizations, as well as scalability due to its full compatibility with 3D parallelism. Results of systematic experiments validate the efficacy of EE-Tuning, confirming that effective early-exit LLM inference can be achieved with a limited training budget. In hope of making early-exit LLMs accessible to the community, we release the source code of our implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑