arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.00625cs.LG
收藏

超越效率:资源高效大型语言模型的系统综述

Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

Emory University(埃默里大学);University of Virginia(弗吉尼亚大学);Pennsylvania State University(宾夕法尼亚州立大学)

Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, Xinyuan Song, Carl Yang, Yue Cheng, Liang Zhao

AI总结:

该综述系统梳理了提升大型语言模型在计算、内存、能源、财务和网络等方面资源效率的技术,覆盖模型全生命周期,并提出标准化评估体系,为可持续高效LLM开发提供参考。

摘要:

大型语言模型(LLMs)这一迅速发展的领域,以OpenAI的ChatGPT等复杂模型为代表,标志着人工智能的重大进步。然而,这些模型在计算、内存、能源和财务资源方面带来了高消耗的重大挑战,尤其是在资源能力有限的环境中。本综述旨在通过回顾一系列旨在提升LLMs资源效率的技术来系统性地应对这些挑战。我们根据优化重点对方法进行分类:计算、内存、能源、财务和网络资源,以及它们在LLM生命周期各个阶段的适用性,包括架构设计、预训练、微调和系统设计。此外,本综述引入了一种按具体资源类型对资源效率技术进行的细致分类,揭示了各种资源与相应优化技术之间错综复杂的关系和映射。还提出了一套标准化的评估指标和数据集,以促进不同模型和技术之间的一致且公平的比较。通过提供当前最先进(sota)的全面概述并识别开放的研究方向,本综述为研究人员和实践者提供了基础性参考,帮助他们在快速发展的环境中开发更可持续、更高效的LLMs。

英文摘要:

The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑