The Cost of AI-Assisted Coding: An Extensive Investigation of Energy vs. Accuracy in Language Models
Utrecht University(乌得勒支大学);University of Twente(特温特大学);Fraunhofer IESE(弗劳恩霍夫IESE研究所)
Negar Alizadeh, Boris Belchev, Nishant Saurabh, Patricia Kelbert, Fernando Castor
备注:
Extended journal version of arXiv:2412.00329v2, originally published in the MSR 2025 conference proceedings. This version includes additional models, experiments, and analyses. Currently under review
Developers increasingly rely on generative AI-based coding assistants such as ChatGPT and GitHub Copilot in their workflows. While many such tools are accessible via remote APIs, data privacy and security concerns drive client organizations toward locally deployed language models. This study examines the accuracy-energy trade-off in local LLM deployment, aiming to provide valuable insights to help developers make informed decisions when selecting a language model. We evaluated 26 LLM families (including both Mixture of Experts and dense architectures) across common software development tasks on two hardware configurations: a commodity GPU and a high-performance AI-specific GPU. As local LLM deployment demands powerful infrastructure investment, which might not be affordable for everyone, we consider both full-precision and quantized variants. Our results demonstrate that larger models with higher energy requirements do not consistently yield proportional accuracy gains. Moreover, quantized large models typically outperform full-precision medium-sized models in both efficiency and accuracy. We also find that no single model excels across all software development task types. Finally, the number of active parameters, output length, and quantization level jointly explain over 90% of the variance in inference energy consumption across all tasks. The results confirm that energy consumption scales with the interaction between output length and model size, consistent with the expected O(np) behavior, while prompt size has a negligible impact.
Ten Simple Rules for AI-Assisted Coding in Science
Yonsei University(延世大学);Stanford University(斯坦福大学);Princeton University(普林斯顿大学);University of Science and Technology of China(中国科学技术大学);University of California, Irvine(加州大学尔湾分校)
Agentic AI-assisted coding offers a unique opportunity to instill epistemic grounding during software development
Center for Proteomics and Metabolomics, Leiden University Medical Center(蛋白质组学与代谢组学中心,莱顿大学医学中心);National Institute of Standards and Technology - Charleston(国家标准与技术研究院-查尔斯顿)
Tokalator: A Context Engineering Toolkit for Artificial Intelligence Coding Assistants
Kariyer.net, R&D Center(Kariyer.net 研发中心);Stanford University and iLab, Stanford, CA, U.S.A.(斯坦福大学及iLab);Centre Tecnològic de Telecomunicacions de Catalunya (CTTC/CERCA)(加泰罗尼亚电信技术中心)
The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution
City University of Hong Kong (Dongguan)(香港城市大学(东莞));The Hong Kong Polytechnic University(香港理工大学);The University of Hong Kong(香港大学);The Hong Kong University of Science and Technology(香港科技大学)
ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
Shanghai Jiao Tong University, Shanghai, China(上海交通大学);Beijing Institute of Technology, Beijing, China(北京理工大学);University of California, Merced, CA, USA(加州大学默塞德分校);Shanghai Innovation Institute, Shanghai, China(上海创新研究院)