arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.04088cs.DC
收藏

迈向可持续的边缘集群大语言模型推理

Toward Sustainability-Aware LLM Inference on Edge Clusters

University of Innsbruck(因斯布鲁克大学);University of Klagenfurt(克拉根福大学)

Kolichala Rajashekar, Nafiseh Sharghivand, Radu Prodan, Reza Farahani

AI总结:

本文提出了一种针对边缘集群的可持续大语言模型推理方法,通过碳和延迟感知的路由策略平衡推理延迟和碳足迹。

摘要:

大型语言模型(LLMs)需要大量的计算资源,导致显著的碳排放和运营成本。尽管训练是能源密集型的,但长期的环境负担来自于推理,这被全球庞大的查询量放大。基于云的推理提供了可扩展性,但由于集中处理和持续的数据传输导致延迟和带宽限制。边缘集群可以通过实现本地执行来缓解这些限制,但它们在性能、能效和设备限制之间面临权衡。本文提出了一种针对边缘集群的可持续大语言模型推理方法,包括NVIDIA Jetson Orin NX(8GB)和Nvidia Ada 2000(16GB)设备。其目标是通过碳和延迟感知的路由策略平衡推理延迟和碳足迹,通过在不同提示和批量(即一组提示)配置中对能耗和执行时间进行实证基准测试来指导。我们比较了基线贪心策略与基于碳和延迟的提示路由策略,基于基准测试信息将提示路由到特定硬件。实验评估显示,四个提示的批量大小在吞吐量、能效之间达到了平衡,而更大的批量则有风险导致GPU内存饱和。

英文摘要:

Large language models (LLMs) require substantial computational resources, leading to significant carbon emissions and operational costs. Although training is energy-intensive, the long-term environmental burden arises from inference, amplified by the massive global query volume. Cloud-based inference offers scalability but suffers from latency and bandwidth constraints due to centralized processing and continuous data transfer. Edge clusters instead can mitigate these limitations by enabling localized execution, yet they face trade-offs between performance, energy efficiency, and device constraints. This short paper presents a sustainability-aware LLM inference for edge clusters comprising NVIDIA Jetson Orin NX (8GB) and Nvidia Ada 2000 (16GB) devices. It aims to balance inference latency and carbon footprint through carbon- and latency-aware routing strategies, guided by empirical benchmarking of energy consumption and execution time across diverse prompts and batch (i.e., group of prompts) configurations. We compared baseline greedy strategies to carbon-aware and latency-aware strategies in prompt routing to specific hardware based on benchmarking information. Experimental evaluation shows that a batch size of four prompts achieves a trade-off between throughput, energy efficiency, while larger batches risk GPU memory saturation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑