大型语言模型(LLMs)需要大量的计算资源,导致显著的碳排放和运营成本。尽管训练是能源密集型的,但长期的环境负担来自于推理,这被全球庞大的查询量放大。基于云的推理提供了可扩展性,但由于集中处理和持续的数据传输导致延迟和带宽限制。边缘集群可以通过实现本地执行来缓解这些限制,但它们在性能、能效和设备限制之间面临权衡。本文提出了一种针对边缘集群的可持续大语言模型推理方法,包括NVIDIA Jetson Orin NX(8GB)和Nvidia Ada 2000(16GB)设备。其目标是通过碳和延迟感知的路由策略平衡推理延迟和碳足迹,通过在不同提示和批量(即一组提示)配置中对能耗和执行时间进行实证基准测试来指导。我们比较了基线贪心策略与基于碳和延迟的提示路由策略,基于基准测试信息将提示路由到特定硬件。实验评估显示,四个提示的批量大小在吞吐量、能效之间达到了平衡,而更大的批量则有风险导致GPU内存饱和。
英文摘要:
Large language models (LLMs) require substantial computational resources, leading to significant carbon emissions and operational costs. Although training is energy-intensive, the long-term environmental burden arises from inference, amplified by the massive global query volume. Cloud-based inference offers scalability but suffers from latency and bandwidth constraints due to centralized processing and continuous data transfer. Edge clusters instead can mitigate these limitations by enabling localized execution, yet they face trade-offs between performance, energy efficiency, and device constraints. This short paper presents a sustainability-aware LLM inference for edge clusters comprising NVIDIA Jetson Orin NX (8GB) and Nvidia Ada 2000 (16GB) devices. It aims to balance inference latency and carbon footprint through carbon- and latency-aware routing strategies, guided by empirical benchmarking of energy consumption and execution time across diverse prompts and batch (i.e., group of prompts) configurations. We compared baseline greedy strategies to carbon-aware and latency-aware strategies in prompt routing to specific hardware based on benchmarking information. Experimental evaluation shows that a batch size of four prompts achieves a trade-off between throughput, energy efficiency, while larger batches risk GPU memory saturation.
MoA-Off: Adaptive Heterogeneous Modality-Aware Offloading with Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Institute of AI for Industries(产业智能研究院);University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室)
MetaFed: Advancing Privacy, Performance, and Sustainability in Federated Metaverse Systems
Department of Computer Engineering(计算机工程系);Faculty of Engineering(工程学院);Kırıkkale University(Kirikkkale 大学);Natural Sciences, Sabancı University(自然科学,萨班奇大学)
Energy-Aware Agentic Orchestration for Multi-Site Service Deployment
Federal University of Viçosa (UFV)(维索萨联邦大学);Federal University of Uberlândia (UFU)(乌贝兰迪亚联邦大学);Instituto de Telecomunicações(电信研究所);Universidade de Aveiro (UA)(阿威罗大学);University of São Paulo (USP)(圣保罗大学);Universitat Politecnica de València (UPV)(瓦伦西亚理工大学);University of Minho (UMinho)(米尼奥大学)
Towards Energy-Aware Federated Traffic Prediction for Cellular Networks
Democritus University of Thrace(色雷斯德谟克利特大学);Imperial College London(伦敦帝国理工学院);Nokia Bell Labs(诺基亚贝尔实验室);Centre Tecnològic de Telecomunicacions de Catalunya(加泰罗尼亚电信技术中心);IMT Atlantique(大西洋国立高等矿业电信学校);Inria(法国国家信息与自动化研究所)