大型语言模型训练与推理的系统性能与成本建模
System-performance and cost modeling of Large Language Model training and inference
AI总结：
本文提出一种融合 Flash Attention、MoE、5D 并行和 chiplet 成本模型的 LLM 训练与推理性能-成本建模方法，用于分析不同系统架构配置下的性能-成本权衡，指导未来计算系统设计与软硬件协同开发。
摘要：
基于 Transformer 架构的大型语言模型（LLMs）凭借其卓越的可扩展性和适应性，已经深刻变革了人工智能、科学和工程领域的众多方向。然而，LLM 规模与复杂度的指数级增长已经超过了计算能力、内存带宽、网络性能和成本效率的进步速度，给其在分布式系统上的可扩展性带来了重大挑战。为了应对这些限制，文献中已经提出了替代模型架构、优化策略、面向通信的网络拓扑以及新型系统设计方法。本文提出了一种面向 LLM 训练和推理的性能-成本建模方法，该方法将最先进的计算技术与内存优化以及最新的通信技术相结合。在解析性能模型的基础上，我们的方法融合了 Flash Attention 技术和 Mixture of Experts 模型等近期创新，以解决内存带宽和计算瓶颈问题。该方法还考虑了不同网络拓扑以及基于 5D 并行性的拓扑专用通信算法的影响。该框架还集成了 chiplet 成本模型。所提出的建模方法为指导未来计算系统设计提供了有价值的见解，并促进硬件-软件协同开发，尤其是因为它能够分析各种系统架构配置下的性能-成本权衡。
英文摘要：
Large language models (LLMs), based on transformer architectures, have revolutionized numerous domains within artificial intelligence, science, and engineering due to their exceptional scalability and adaptability. However, the exponential growth in LLM size and complexity has outpaced advancements in compute capacity, memory bandwidth, network performance, and cost efficiency, posing significant challenges to their scalability on distributed systems. To address these limitations, alternative model architectures, optimization strategies, communication-aware network topologies, and novel system design approaches have been proposed in literature. This paper introduces a performance-cost modeling methodology for LLM training and inference that integrates state-of-the-art compute techniques with memory optimizations, and latest communication techniques. Building on an analytical performance model, our approach incorporates recent innovations such as the flash attention technique and mixture of experts models to address the memory bandwidth and compute bottlenecks. It also considers the impact of different network topologies and topology-specific communication algorithms with 5D parallellism. The framework also integrates a chiplet cost model. The proposed modeling methodology provides valuable insights to guide future compute system design and facilitates hardware-software co-development, in particular due to its ability to analyze performance-cost trade-offs for various system architectural configurations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
