使用大型语言模型评估源代码质量：一项比较研究
Evaluating Source Code Quality with Large Language Models: a comparative study
AI总结：
本文比较了GPT 3.5 Turbo和GPT 4o与SonarQube在Java开源项目代码质量评估上的表现，发现GPT 3.5 Turbo与Sonar度量相关但GPT 4o结果存在分歧，展示了LLM用于代码质量评估的潜力与局限。
摘要：
代码质量是由多种度量指标构成的属性，例如复杂度、可读性、可测试性、互操作性、可复用性以及良好或不良实践的使用等。静态代码分析工具旨在测量一组属性以评估代码质量。然而，某些质量属性只能由人在代码评审活动中衡量，可读性便是一个例子。鉴于大型语言模型（LLM）的自然语言文本处理能力，我们假设LLM能够评估代码质量，包括当前无法自动化的属性。本文旨在描述并分析将LLM用作静态分析工具、评估代码整体质量所获得的结果。我们将LLM与SonarQube软件及其Maintainability（可维护性）度量指标在两个开源软件（OSS）Java项目上获得的结果进行了比较，其中一个项目的Maintainability Rating为A，另一个为B。共分析了1,641个类，并在两个模型版本上比较了结果：GPT 3.5 Turbo和GPT 4o。我们证明了GPT 3.5 Turbo LLM具有评估代码质量的能力，并显示出与Sonar度量指标的相关性。然而，与SonarQube相比，LLM所测量的内容在某些具体方面存在差异。GPT 4o版本并未呈现相同的结果，它与前一个模型及Sonar产生分歧，对被评为较低质量的代码赋予了较高分类。本研究展示了LLM在评估代码质量方面的潜力。然而，仍需进一步研究以调查其局限性，例如LLM的成本、输出可变性，并探索传统静态分析工具未测量的质量特征。
英文摘要：
Code quality is an attribute composed of various metrics, such as complexity, readability, testability, interoperability, reusability, and the use of good or bad practices, among others. Static code analysis tools aim to measure a set of attributes to assess code quality. However, some quality attributes can only be measured by humans in code review activities, readability being an example. Given their natural language text processing capability, we hypothesize that a Large Language Model (LLM) could evaluate the quality of code, including attributes currently not automatable. This paper aims to describe and analyze the results obtained using LLMs as a static analysis tool, evaluating the overall quality of code. We compared the LLM with the results obtained with the SonarQube software and its Maintainability metric for two Open Source Software (OSS) Java projects, one with Maintainability Rating A and the other B. A total of 1,641 classes were analyzed, comparing the results in two versions of models: GPT 3.5 Turbo and GPT 4o. We demonstrated that the GPT 3.5 Turbo LLM has the ability to evaluate code quality, showing a correlation with Sonar's metrics. However, there are specific aspects that differ in what the LLM measures compared to SonarQube. The GPT 4o version did not present the same results, diverging from the previous model and Sonar by assigning a high classification to codes that were assessed as lower quality. This study demonstrates the potential of LLMs in evaluating code quality. However, further research is necessary to investigate limitations such as LLM's cost, variability of outputs and explore quality characteristics not measured by traditional static analysis tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
