The rapid evolution and widespread adoption of generative large language models (LLMs) have made them a pivotal workload in various applications. Today, LLM inference clusters receive a large number of queries with strict Service Level Objectives (SLOs). To achieve the desired performance, these models execute on power-hungry GPUs causing the inference clusters to consume large amount of energy and, consequently, result in excessive carbon emissions. Fortunately, we find that there is a great opportunity to exploit the heterogeneity in inference compute properties and fluctuations in inference workloads, to significantly improve energy-efficiency. However, such a diverse and dynamic environment creates a large search-space where different system configurations (e.g., number of instances, model parallelism, and GPU frequency) translate into different energy-performance trade-offs. To address these challenges, we propose DynamoLLM, the first energy-management framework for LLM inference environments. DynamoLLM automatically and dynamically reconfigures the inference cluster to optimize for energy and cost of LLM serving under the service's performance SLOs. We show that at a service-level, DynamoLLM conserves 53% energy and 38% operational carbon emissions, and reduces 61% cost to the customer, while meeting the latency SLOs.
Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机学院)
OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming
Key Lab of High Confidence Software Technologies (Peking University)(高可信软件技术教育部重点实验室(北京大学));State Key Laboratory of Networking and Switching Technology (BUPT)(网络与交换技术国家重点实验室(北京邮电大学))
From Consumption to Reflection: Designing Human-AI Relations for Stable Reasoning
Faculty of Medicine, Lund University(吕勒欧大学医学院);Department of Economics, Lund University School of Economics and Management(吕勒欧大学经济学与管理学院经济系);Department of Health Services Research and Policy, London School of Hygiene & Tropical Medicine(伦敦卫生与热带医学学院健康服务研究与政策系)
Designing explainable artificial intelligence with active inference: A framework for transparent introspection and decision-making
VERSES Research Lab(VERSES研究实验室);Université du Québec à Montréal(魁北克大学蒙特利尔分校);Humboldt-Universität zu Berlin(柏林洪堡大学);Macquarie University(麦考瑞大学);University College London(伦敦大学学院)
Digital Twin Enabled Site Specific Channel Precoding: Over the Air CIR Inference
Department of EECS, Howard University(霍华德大学电子工程与计算机科学系);Department of Electrical and Computer Engineering, Université Laval(拉瓦尔大学电子与计算机工程系);DeepSig Inc.(DeepSig公司)
Designing Ecosystems of Intelligence from First Principles
VERSES AI Research Lab(VERSES人工智能研究实验室);University College London(伦敦大学学院);Monash University(莫纳什大学);University of Sussex(萨塞克斯大学);Université du Québec à Montréal(魁北克大学蒙特利尔分校);University of Oxford(牛津大学);Max Planck Institute of Animal Behavior(马克斯·普朗克动物行为研究所);University of Konstanz(康斯坦茨大学);Network Science Institute(网络科学研究所)