实现大型语言模型的峰值性能：一项系统综述
Achieving Peak Performance for Large Language Models: A Systematic Review
AI总结：
本文遵循PRISMA对65篇文献进行系统综述，从训练、推理和系统服务三方面梳理LLM优化加速方法，并通过案例说明如何在不损失精度下缓解资源限制。
摘要：
近年来，大型语言模型（LLM）在自然语言处理（NLP）领域取得了显著成功。LLM需要极其大量的参数才能获得高性能。随着模型增长至万亿参数规模，计算和内存成本大幅增加。这使得许多研究人员难以获得训练或应用这些模型所需的资源。优化LLM性能主要涉及两种途径：针对特定任务微调预训练模型以实现最先进性能，以及在保持相似性能的同时降低成本或缩短训练时间。本文呈现了一项遵循PRISMA（系统综述与荟萃分析首选报告项目）声明的系统文献综述（SLR）。我们从5个数据库检索了2017年至2023年12月的983篇出版物，并对其中65篇进行了综述。本研究提出了在不牺牲准确性的前提下优化和加速LLM并取得前沿结果的方法。我们首先概述语言建模的发展，随后详细解释常用框架和库，并基于LLM训练、LLM推理和系统服务三个类别，提出用于改进和加速LLM的分类体系。然后，我们深入探讨近期的优化与加速策略，例如训练优化、硬件优化、可扩展性和可靠性，并对这些策略进行分类和归纳。最后，我们对每个类别和策略进行深入比较，并提供两个关于优化模型训练和提升推理效率的案例研究。这些案例研究展示了在保持性能的同时应对LLM资源限制的实用方法。
英文摘要：
In recent years, large language models (LLMs) have achieved remarkable success in natural language processing (NLP). LLMs require an extreme amount of parameters to attain high performance. As models grow into the trillion-parameter range, computational and memory costs increase significantly. This makes it difficult for many researchers to access the resources needed to train or apply these models. Optimizing LLM performance involves two main approaches: fine-tuning pre-trained models for specific tasks to achieve state-of-the-art performance, and reducing costs or improving training time while maintaining similar performance. This paper presents a systematic literature review (SLR) following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We reviewed 65 publications out of 983 from 2017 to December 2023, retrieved from 5 databases. The study presents methods to optimize and accelerate LLMs while achieving cutting-edge results without sacrificing accuracy. We begin with an overview of the development of language modeling, followed by a detailed explanation of commonly used frameworks and libraries, and a taxonomy for improving and speeding up LLMs based on three classes: LLM training, LLM inference, and system serving. We then delve into recent optimization and acceleration strategies such as training optimization, hardware optimization, scalability and reliability, accompanied by the taxonomy and categorization of these strategies. Finally, we provide an in-depth comparison of each class and strategy, with two case studies on optimizing model training and enhancing inference efficiency. These case studies showcase practical approaches to address LLM resource limitations while maintaining performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
