Large language model (LLM) services now answer billions of queries per day, and industry reports show that inference, not training, accounts for more than 90% of total power consumption. However, existing benchmarks focus on either training/fine-tuning or performance of inference and provide little support for power consumption measurement and analysis of inference. We introduce TokenPowerBench, the first lightweight and extensible benchmark designed for LLM-inference power consumption studies. The benchmark combines (i) a declarative configuration interface covering model choice, prompt set, and inference engine, (ii) a measurement layer that captures GPU-, node-, and system-level power without specialized power meters, and (iii) a phase-aligned metrics pipeline that attributes energy to the prefill and decode stages of every request. These elements make it straight-forward to explore the power consumed by an LLM inference run; furthermore, by varying batch size, context length, parallelism strategy and quantization, users can quickly assess how each setting affects joules per token and other energy-efficiency metrics. We evaluate TokenPowerBench on four of the most widely used model series (Llama, Falcon, Qwen, and Mistral). Our experiments cover from 1 billion parameters up to the frontier-scale Llama3-405B model. Furthermore, we release TokenPowerBench as open source to help users to measure power consumption, forecast operating expenses, and meet sustainability targets when deploying LLM services.
EnergyStar++: Towards more accurate and explanatory building energy benchmarking
Berkeley Education Alliance for Research in Singapore(新加坡伯克利教育研究联盟);University of California, Berkeley(加州大学伯克利分校);National University of Singapore(新加坡国立大学)
Benchmarking Artificial Intelligence Models for Daily Coastal Hypoxia Forecasting
Department of Environmental Sciences, Louisiana State University(环境科学系,路易斯安那州立大学);Center for Computation and Technology, Louisiana State University(计算与技术中心,路易斯安那州立大学);Department of Computer Science, Louisiana State University(计算机科学系,路易斯安那州立大学);Department of Computer Science, Furman University(计算机科学系,富兰克林大学);Department of Oceanography and Coastal Sciences, Louisiana State University(海洋学与沿海科学系,路易斯安那州立大学)
PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments
University of Science and Technology of China(中国科学技术大学);City University of Hong Kong(香港城市大学);Northeastern University(东北大学);MemTensor (Shanghai) Technology Co., Ltd.(MemTensor(上海)科技有限公司)
Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
SINTEF(辛泰夫(挪威研究机构));Singapore Management University(新加坡管理大学);Oslo Metropolitan University(奥斯陆城市大学);Kristiania University of Applied Sciences(克里斯蒂安尼亚应用科技大学)
本研究在Raspberry Pi 4上评估了28个量化LLM在能效、推理性能和准确率上的权衡,通过硬件级能耗分析与多数据集基准测试,为资源受限环境下的可持续LLM部署提供了可操作的见解。
Benchmarking energy consumption and latency for neuromorphic computing in condensed matter and particle physics
University of Twente(特文特大学);Radboud University(拉德堡德大学);IBM Research Europe - Zürich(IBM欧洲研究院-苏黎世);Nikhef(荷兰国家亚原子物理研究所);Eidgenössische Technische Hochschule Zürich(苏黎世联邦理工学院);SURF Cooperation(SURF合作组织);University of Valencia-CSIC(瓦伦西亚大学-西班牙国家研究委员会)
Fine-tuning LLaMA 2 interference: a comparative study of language implementations for optimal efficiency
Synergy University(协同大学);BGC Trust University(BGC信托大学);Chittagong University of Engineering and Technology(吉大港工程技术大学);Samarkand State University(撒马尔罕国立大学);Military Institute of Science and Technology (MIST)(科技军事学院);Sonargaon University(索纳冈大学)
Automated Machine Learning: A Case Study on Non-Intrusive Appliance Load Monitoring
Department of Computer Science University of Colorado Colorado Springs(计算机科学系,科罗拉多州立大学);School of Computation, Information, and Technology (CIT) Technical University of Munich(计算、信息与技术学院(CIT),慕尼黑技术大学);School of CIT Technical University of Munich(计算、信息与技术学院,慕尼黑技术大学);Software AG Munich, Germany(软件AG,德国慕尼黑);School of CIT and MDSI Technical University of Munich(计算、信息与技术学院和MDSI,慕尼黑技术大学)