arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.00329cs.SE
收藏

软件开发任务中的语言模型:能耗与准确率的实验分析

The Cost of AI-Assisted Coding: An Extensive Investigation of Energy vs. Accuracy in Language Models

Utrecht University(乌得勒支大学);University of Twente(特温特大学);Fraunhofer IESE(弗劳恩霍夫IESE研究所)

Negar Alizadeh, Boris Belchev, Nishant Saurabh, Patricia Kelbert, Fernando Castor

AI总结:

本研究针对软件开发场景下本地部署大语言模型的需求,在两类GPU上测试18个LLM家族全精度与量化版本的准确率与能耗,发现大模型高能耗不必然提准、量化大模型优于全精度中模型,且无模型适配所有开发任务,为开发者选型提供参考。

摘要:

基于生成式AI的编程助手(如ChatGPT、Github Copilot)的应用已成为当代软件开发的现实。这类工具多以远程API形式提供,使用第三方API会给客户企业带来数据隐私与安全顾虑,这也推动了本地部署语言模型的应用。本研究探究模型准确率与能耗之间的权衡关系,旨在为开发者选择语言模型时提供有价值的参考,助力其做出明智决策。我们在两种真实基础设施(消费级GPU与高性能专用AI GPU)上,测试了18个大语言模型(LLM)家族在典型软件开发任务中的表现。考虑到本地部署LLM需要高性能基础设施,并非所有主体都能负担,我们同时评估了全精度模型与量化模型。研究结果表明,使用能耗预算更高的大型LLM并不总能带来准确率的显著提升;此外,大型模型的量化版本通常比中型模型的全精度版本具备更优的效率与准确率。除此之外,没有任何单一模型能适配所有类型的软件开发任务。

英文摘要:

Developers increasingly rely on generative AI-based coding assistants such as ChatGPT and GitHub Copilot in their workflows. While many such tools are accessible via remote APIs, data privacy and security concerns drive client organizations toward locally deployed language models. This study examines the accuracy-energy trade-off in local LLM deployment, aiming to provide valuable insights to help developers make informed decisions when selecting a language model. We evaluated 26 LLM families (including both Mixture of Experts and dense architectures) across common software development tasks on two hardware configurations: a commodity GPU and a high-performance AI-specific GPU. As local LLM deployment demands powerful infrastructure investment, which might not be affordable for everyone, we consider both full-precision and quantized variants. Our results demonstrate that larger models with higher energy requirements do not consistently yield proportional accuracy gains. Moreover, quantized large models typically outperform full-precision medium-sized models in both efficiency and accuracy. We also find that no single model excels across all software development task types. Finally, the number of active parameters, output length, and quantization level jointly explain over 90% of the variance in inference energy consumption across all tasks. The results confirm that energy consumption scales with the interaction between output length and model size, consistent with the expected O(np) behavior, while prompt size has a negligible impact.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑