智能但代价高昂?在功能准确性和能源效率上对大型语言模型进行基准测试
Smart but Costly? Benchmarking LLMs on Functional Accuracy and Energy Efficiency
AI总结:
针对现有方法缺乏评估代码语言模型(CLMs)准确性-能源权衡框架的问题,提出BRACE框架及CIRC、OTER两种评级方法,基准测试22个模型,发现模型在代码摘要任务表现更好、大小对评级无显著影响,助力从业者平衡可持续性与任务需求选择模型。
摘要:
人工智能技术的快速发展及其在软件开发中的加速应用,使得对其环境影响和功能正确性进行系统评估成为必要。虽然先前的研究已经考察了大型语言模型的可持续性,但现有方法缺乏评估代码语言模型(CLMs)准确性-能源权衡的系统框架。在本文中,我们提出了一个名为BRACE的框架,用于在能源效率和功能正确性(称为准确性)的统一尺度上对CLMs进行基准测试。我们在代码生成和摘要任务上对22个最先进的模型进行了基准测试,提出了两种评级方法:同心增量评级圈(CIRC)和观察到期望评级(OTER)。CIRC提供基于欧几里得的确定性排名,具有对异常值鲁棒的静态权衡;OTER提供具有动态权衡的趋势感知评估,捕捉能源和准确性之间的复杂相关性,每种方法都提供独特的视角并以独特的方式解决问题。这些评级方法使我们能够在1-5的尺度上对大型语言模型进行评级,反映它们在能源效率和功能正确性方面的综合能力。我们的分析表明,模型在代码摘要任务中通常表现更好,因为它们不需要生成基于语法且句法正确的输出。此外,我们发现模型的大小对其评级没有显著影响,这表明如果模型有效地利用其参数,它们可以在这些尺度上获得更高的排名。所提出的BRACE框架使从业者能够做出基于证据的模型选择,平衡可持续性与任务要求,并根据部署优先级指导评级选择——CIRC用于确定性比较,OTER用于趋势感知评估。
英文摘要:
The rapid advancement of AI technologies and their accelerated adoption in software development necessitates a systematic evaluation of their environmental impact alongside functional correctness. While prior studies have examined sustainability in large language models, existing approaches lack systematic frameworks for evaluating accuracy-energy trade-offs in Code Language Models (CLMs). In this paper, we present a framework, BRACE, to benchmark CLMs on a unified scale of energy efficiency and functional correctness (referred to as accuracy). We benchmark 22 state-of-the-art models on code generation and summarization tasks, proposing two rating methods: Concentric Incremental Rating Circles (CIRC) and Observation to Expectation Rating (OTER). CIRC provides deterministic Euclidean-based rankings with static trade-offs that are robust to outliers, and OTER offers trend-aware evaluation with dynamic trade-offs that capture the complex correlation between energy and accuracy, each offering a distinct perspective and addressing the problem in a unique way. These rating methods enable us to rate LLMs on a 1-5 scale reflecting their combined capabilities in terms of energy efficiency and functional correctness. Our analysis reveals models generally perform better in the code summarization tasks as they are not enforced to generate a grammar-based and syntactically correct output. Also, we find that models' size does not have a significant impact on their ratings, indicating that if models utilize their parameters efficiently, they can be ranked higher on these scales. The proposed BRACE framework empowers practitioners to make evidence-based model selections that balance sustainability with task requirements, guiding rating choice -- CIRC for deterministic comparisons or OTER for trend-aware evaluation -- based on deployment priorities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。