Theseus：探索面向大语言模型的高效晶圆级芯片设计
Theseus: Exploring Efficient Wafer-Scale Chip Design for Large Language Models
AI总结：
本文提出 Theseus 框架，通过多保真度贝叶斯优化高效探索晶圆级芯片设计空间，在 LLM 训练和推理的性能与功耗上显著优于 GPU 集群和现有 WSC 设计。
摘要：
大语言模型（LLM）的出现使计算吞吐量、内存容量和通信带宽的需求呈指数级增长。这种需求增长已显著超过相应芯片设计的改进速度。随着制造与集成技术的进步，设计人员一直在开发晶圆级芯片（Wafer-Scale Chips，WSCs），以在单芯片层面扩大并挖掘计算密度、内存容量和通信带宽的极限。现有解决方案已经证明了 WSC 相对于传统设计的显著优势，显示出有效支持 LLM 工作负载的潜力。尽管有这些好处，但由于设计空间巨大且复杂、评估方法耗时以及探索策略效率低下，针对 LLM 的 WSC 早期设计空间探索是一项关键但具有挑战性的任务。为应对这些挑战，我们提出了 Theseus，一个面向 LLM 的高效 WSC 设计空间探索框架。我们考虑 WSC 的独特特性，构建了带有各种约束的 WSC 设计空间。我们为基于 NoC 的大规模 WSC 提出了高效评估方法，并引入多保真度贝叶斯优化以高效探索设计空间。评估结果证明了 Theseus 的效率：搜索到的帕累托最优结果在 LLM 训练性能上比 GPU 集群和现有 WSC 设计最多高出 62.8%/73.7%，功耗最多降低 38.6%/42.4%；在推理任务中，性能和功耗分别最多提升 23.2 倍和 15.7 倍。此外，我们进行了案例研究以解决 WSC 中的设计权衡问题，并为促进面向 LLM 的 WSC 设计提供了见解。
英文摘要：
The emergence of the large language model~(LLM) poses an exponential growth of demand for computation throughput, memory capacity, and communication bandwidth. Such a demand growth has significantly surpassed the improvement of corresponding chip designs. With the advancement of fabrication and integration technologies, designers have been developing Wafer-Scale Chips~(WSCs) to scale up and exploit the limits of computation density, memory capacity, and communication bandwidth at the level of a single chip. Existing solutions have demonstrated the significant advantages of WSCs over traditional designs, showing potential to effectively support LLM workloads. Despite the benefits, exploring the early-stage design space of WSCs for LLMs is a crucial yet challenging task due to the enormous and complicated design space, time-consuming evaluation methods, and inefficient exploration strategies. To address these challenges, we propose Theseus, an efficient WSC design space exploration framework for LLMs. We construct the design space of WSCs with various constraints considering the unique characteristics of WSCs. We propose efficient evaluation methodologies for large-scale NoC-based WSCs and introduce multi-fidelity Bayesian optimization to efficiently explore the design space. Evaluation results demonstrate the efficiency of Theseus that the searched Pareto optimal results outperform GPU cluster and existing WSC designs by up to 62.8\%/73.7\% in performance and 38.6\%/42.4\% in power consumption for LLM training, while improving up to 23.2$\times$ and 15.7$\times$ for the performance and power of inference tasks. Furthermore, we conduct case studies to address the design tradeoffs in WSCs and provide insights to facilitate WSC designs for LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
