ECHO-LLaMA：面向高性能LLaMA训练的高效缓存机制
ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training
AI总结：
ECHO-LLaMA通过跨层共享KV缓存降低计算复杂度，在保持性能的同时显著提升LLaMA训练吞吐量、MFU并降低损失，为大模型训练提供高效可扩展方案。
摘要：
本文介绍了ECHO-LLaMA，一种高效的LLaMA架构，旨在提升LLaMA架构的训练速度和推理吞吐量，同时保持其学习能力。ECHO-LLaMA将LLaMA模型转换为在特定层之间共享KV缓存，显著降低了KV计算复杂度，同时保持或提升了语言性能。实验结果表明，在训练相同数量的token时，ECHO-LLaMA的每秒token吞吐量最高可提升77%，模型FLOPs利用率（MFU）最高可提升16%，损失最多可降低14%。此外，在1.1B模型上，与基线相比，ECHO-LLaMA的测试时吞吐量提高了约7%。通过引入一种计算高效的适应机制，ECHO-LLaMA为大语言模型的预训练和微调提供了一种可扩展且经济高效的解决方案，能够在不影响性能的前提下实现更快、更资源高效的训练。
英文摘要：
This paper introduces ECHO-LLaMA, an efficient LLaMA architecture designed to improve both the training speed and inference throughput of LLaMA architectures while maintaining its learning capacity. ECHO-LLaMA transforms LLaMA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. Experimental results demonstrate that ECHO-LLaMA achieves up to 77\% higher token-per-second throughput during training, up to 16\% higher Model FLOPs Utilization (MFU), and up to 14\% lower loss when trained on an equal number of tokens. Furthermore, on the 1.1B model, ECHO-LLaMA delivers approximately 7\% higher test-time throughput compared to the baseline. By introducing a computationally efficient adaptation mechanism, ECHO-LLaMA offers a scalable and cost-effective solution for pretraining and finetuning large language models, enabling faster and more resource-efficient training without compromising performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
