关于源代码语言模型的共享与能源使用的探索性文献研究
An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code
AI总结：
该研究通过滚雪球式文献调查分析了软件工程领域源代码语言模型的共享与能耗透明度，发现仅27%的论文共享训练产物且40%未共享源代码或模型，并据此提出共享代码、训练产物及训练能耗信息的可持续复现建议。
摘要：
在源代码上训练的大型语言模型可以支持多种软件开发任务，例如代码推荐和程序修复。用于训练此类模型的大量数据有助于提升模型性能。然而，数据和模型的规模导致训练时间长且能耗高。虽然发布源代码可以实现可复现性，但如果模型未被共享，用户就需要重复昂贵的训练过程。本研究的主要目标是调查那些为软件工程（SE）任务训练语言模型的出版物是否共享源代码和训练产物。第二个目标是分析训练能源使用方面的透明度。我们执行了一项基于滚雪球法的文献搜索，以查找关于源代码语言模型的出版物，并从可持续性的角度分析其可复用性。\n从494篇唯一出版物中，我们识别出293篇使用语言模型解决代码相关任务的相关出版物。其中，27%（293篇中的79篇）提供了可供复用的产物。这些产物的形式可以是针对特定任务设计的工具或IDE插件，也可以是可针对各种下游任务进行微调的任务无关模型。此外，我们收集了关于模型训练所用硬件以及训练时间的信息，这两者共同决定了开发过程的能源消耗。我们发现，当前针对软件工程任务的源代码模型研究在信息和产物共享方面存在不足，被调查的论文中有40%未共享源代码或训练产物。我们建议共享源代码以及训练产物，以实现可持续的可复现性。此外，应共享关于训练时间和硬件配置的全面信息，以提高模型碳足迹的透明度。
英文摘要：
Large language models trained on source code can support a variety of software development tasks, such as code recommendation and program repair. Large amounts of data for training such models benefit the models' performance. However, the size of the data and models results in long training times and high energy consumption. While publishing source code allows for replicability, users need to repeat the expensive training process if models are not shared. The main goal of the study is to investigate if publications that trained language models for software engineering (SE) tasks share source code and trained artifacts. The second goal is to analyze the transparency on training energy usage. We perform a snowballing-based literature search to find publications on language models for source code, and analyze their reusability from a sustainability standpoint. From 494 unique publications, we identified 293 relevant publications that use language models to address code-related tasks. Among them, 27% (79 out of 293) make artifacts available for reuse. This can be in the form of tools or IDE plugins designed for specific tasks or task-agnostic models that can be fine-tuned for a variety of downstream tasks. Moreover, we collect insights on the hardware used for model training, as well as training time, which together determine the energy consumption of the development process. We find that there are deficiencies in the sharing of information and artifacts for current studies on source code models for software engineering tasks, with 40% of the surveyed papers not sharing source code or trained artifacts. We recommend the sharing of source code as well as trained artifacts, to enable sustainable reproducibility. Moreover, comprehensive information on training times and hardware configurations should be shared for transparency on a model's carbon footprint.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
