分布式LLM预训练期间可再生能源削减窗口:可行性研究
Distributed LLM Pretraining During Renewable Curtailment Windows: A Feasibility Study
AI总结:
本研究探讨在可再生能源削减窗口期间利用清洁廉价电力进行分布式LLM预训练的可行性,通过联邦学习框架实现跨地理分布集群的弹性训练,降低碳排放。
摘要:
训练大型语言模型(LLMs)需要大量计算和能源。同时,可再生能源来源经常产生比电网能吸收的更多电力,导致削减,即故意减少清洁发电,否则会浪费。这些时期代表了一个机会:如果训练与削减窗口对齐,LLMs可以在清洁且便宜的电力上进行预训练。本技术报告介绍了一个系统,该系统在区域削减窗口期间跨地理分布的GPU集群执行全参数LLM训练,在站点可用或不可用时弹性地在本地单站点训练和联邦多站点同步之间切换。我们的原型使用Flower联邦学习框架,在三个集群上训练了一个5.61亿参数的transformer模型,其中削减期源自现实世界中的边际碳强度轨迹。初步结果显示,具有削减意识的调度在保持训练质量的同时,将运营排放减少到单站点基准的5-12%。
英文摘要:
Training large language models (LLMs) requires substantial compute and energy. At the same time, renewable energy sources regularly produce more electricity than the grid can absorb, leading to curtailment, the deliberate reduction of clean generation that would otherwise go to waste. These periods represent an opportunity: if training is aligned with curtailment windows, LLMs can be pretrained using electricity that is both clean and cheap. This technical report presents a system that performs full-parameter LLM training across geo-distributed GPU clusters during regional curtailment windows, elastically switching between local single-site training and federated multi-site synchronization as sites become available or unavailable. Our prototype trains a 561M-parameter transformer model across three clusters using the Flower federated learning framework, with curtailment periods derived from real-world marginal carbon intensity traces. Preliminary results show that curtailment-aware scheduling preserves training quality while reducing operational emissions to 5-12% of single-site baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。