arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.19635cs.CL
收藏

GKT:一种面向高效云边协同LLM部署的基于引导的知识转移新框架

GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment

Shanghai Jiao Tong University(上海交通大学);Wuhan University(武汉大学)

Yao Yao, Zuchao Li, Hai Zhao

AI总结:

针对LLM推理慢且加速方法成本高、各自独立的问题,提出无需微调的GKT框架,利用大模型生成引导提示配合小模型输出,实现云边协同下即插即用的高效加速,在GSM8K和CSQA上显著提升准确率与速度。

摘要:

大型语言模型(LLMs)不断增长的规模提升了其生成响应的能力,但代价是推理时间增加和资源需求升高。现有的加速方法主要依赖知识蒸馏,通常需要对如 Llama-7B 等相当大的模型进行微调,这对普通用户构成了挑战。此外,当前用于加速推理和降低成本的技术各自独立运行。为了解决这些问题,我们提出了一种新颖且直观的基于引导的知识转移(GKT)框架。该方法利用较大的 LLM 作为“教师”来创建引导提示,并配以较小的“学生”模型来完成响应。值得注意的是,GKT 无需微调,且不要求教师和学生模型使用相同的词汇表,从而允许进行大规模批量生成以加速处理过程,同时确保用户定制化。GKT 可无缝集成到云边协同架构中,并且具备足够的通用性,可跨各种模型进行即插即用应用。它在效率和可负担性方面均表现出色,堪称一种“廉价且令人愉悦”的解决方案。GKT 在 GSM8K 上实现了最高 14.18% 的准确率提升以及 10.72 倍的加速,在 CSQA 上实现了 14.00% 的准确率提升以及 7.73 倍的加速。当使用 ChatGPT 作为教师模型、Llama2-70B 作为学生模型时,我们可以以 52% 的成本达到 ChatGPT 95.00% 的性能。结果突显了在 GSM8K 和 CSQA 数据集上准确率和处理速度的显著提升,超越了单独使用学生或教师模型的性能。

英文摘要:

The burgeoning size of Large Language Models (LLMs) has led to enhanced capabilities in generating responses, albeit at the expense of increased inference times and elevated resource demands. Existing methods of acceleration, predominantly hinged on knowledge distillation, generally necessitate fine-tuning of considerably large models, such as Llama-7B, posing a challenge for average users. Furthermore, present techniques for expediting inference and reducing costs operate independently. To address these issues, we introduce a novel and intuitive Guidance-based Knowledge Transfer (GKT) framework. This approach leverages a larger LLM as a ''teacher'' to create guidance prompts, paired with a smaller ''student'' model to finalize responses. Remarkably, GKT requires no fine-tuning and doesn't necessitate the teacher and student models to have the same vocabulary, allowing for extensive batch generation to accelerate the process while ensuring user customization. GKT can be seamlessly integrated into cloud-edge collaboration architectures, and is versatile enough for plug-and-play application across various models. It excels in both efficiency and affordability, epitomizing a ''cheap and cheerful'' solution. GKT achieves a maximum accuracy improvement of 14.18%, along with a 10.72 times speed-up on GSM8K and an accuracy improvement of 14.00 % along with a 7.73 times speed-up in CSQA. When utilizing ChatGPT as teacher model and Llama2-70B as the student model, we can achieve 95.00% of ChatGPT's performance at 52% of the cost. The results highlight substantial enhancements in accuracy and processing speed on the GSM8K and CSQA datasets, surpassing the performance of using either the student or teacher models in isolation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑