ALPINE:一种适用于代码语言模型的自适应语言无关剪枝方法
ALPINE: An adaptive language-agnostic pruning method for language models for code
AI总结:
针对代码语言模型计算资源需求高的问题,提出自适应语言无关剪枝技术ALPINE,通过可插拔层实现输入序列自适应压缩,在保持98.1%原始性能的同时,平均减少50%FLOPs、58.1%内存占用,提升28.1%吞吐量,减少44.85%碳排放。
摘要:
代码语言模型在各种软件工程和源代码分析任务中展现出了最先进的性能。然而,其高计算资源需求及由此产生的环境足迹仍是重大挑战。本研究提出ALPINE,一种自适应的、与编程语言无关的剪枝技术,旨在大幅降低这些模型的计算开销。所提出的方法提供一个可插拔层,可与所有基于Transformer的模型集成。通过ALPINE,输入序列在整个处理流程中进行自适应压缩,压缩后大小最多为初始大小的1/3,从而显著降低计算负载。我们在缺陷预测和代码克隆检测两个软件工程任务上,对CodeBERT、GraphCodeBERT和UniXCoder三个语言模型进行实验,结果显示ALPINE平均实现了50%的FLOPs减少、58.1%的内存占用减少和28.1%的吞吐量提升,还减少了高达44.85%的二氧化碳排放。重要的是,它在减少计算资源的同时,保持了原始预测性能的98.1%。这些发现凸显了ALPINE在提高代码语言模型资源效率和可访问性、同时保持其性能方面的潜力,有助于提升软件开发中语言模型应用的整体可持续性。此外,ALPINE实现的显著序列压缩表明,源代码分析语料库中存在冗余和噪声信息。
英文摘要:
Language models of code have demonstrated state-of-the-art performance across various software engineering and source code analysis tasks. However, their demanding computational resource requirements and consequential environmental footprint remain as significant challenges. This work introduces ALPINE, an adaptive programming language-agnostic pruning technique designed to substantially reduce these models' computational overhead. The proposed method offers a pluggable layer that can be integrated with all Transformer-based models. With ALPINE, input sequences undergo adaptive compression throughout the pipeline, reaching a size up to $\times 3$ less their initial size, resulting in significantly reduced computational load. Our experiments on two software engineering tasks, defect prediction and code clone detection across three language models CodeBERT, GraphCodeBERT and UniXCoder show that ALPINE achieves up to a 50% reduction in FLOPs, a 58.1% decrease in memory footprint, and a 28.1% improvement in throughput on average. This led to a reduction in CO2 by up to $44.85$%. Importantly, it achieves the reduction in computation resources while maintaining up to 98.1% of the original predictive performance. These findings highlight the potential of ALPINE in making language models of code more resource-efficient and accessible while preserving their performance, contributing to the overall sustainability of adopting language models in software development. Also, it sheds light on redundant and noisy information in source code analysis corpora, as shown by the substantial sequence compression achieved by ALPINE.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。