arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.10754cs.SE
收藏

幽灵回声揭示:针对人类评估的可维护性度量与机器学习预测基准测试

Ghost Echoes Revealed: Benchmarking Maintainability Metrics and Machine Learning Predictions Against Human Assessments

Markus Borg, Marwa Ezzouhri, Adam Tornhill

AI总结:

本研究基准测试了多种可维护性预测方法,发现CodeScene在准确性上媲美最先进机器学习并优于人类专家,同时提供可操作的代码异味细节,而SonarQube易产生误报,建议采用更准确的度量以提升研究可靠性。

摘要:

随着生成式人工智能预计将增加全球代码量,从人类视角出发的可维护性将变得更加重要。目前已开发出多种方法用于识别最关键的可维护性问题,包括聚合度量和先进的机器学习(ML)模型。本研究对多种可维护性预测方法进行了基准测试,包括最先进的机器学习(SotA ML)、SonarQube的可维护性评级、CodeScene的代码健康度以及微软的可维护性指数。我们的结果表明,CodeScene在准确性上与SotA ML相当,并且优于普通人类专家。重要的是,与SotA ML不同,CodeScene还向最终用户提供可操作的代码异味细节,以修复已识别的问题。最后,建议谨慎使用SonarQube,因为它倾向于产生大量误报。不幸的是,我们的发现对先前仅依赖SonarQube输出建立真实标签的研究的有效性提出了质疑。为了提高未来可维护性和技术债务研究的可靠性,我们建议采用更准确的度量。此外,使用Code Health重新评估先前的研究结果将缓解这一已揭示的有效性威胁。

英文摘要:

As generative AI is expected to increase global code volumes, the importance of maintainability from a human perspective will become even greater. Various methods have been developed to identify the most important maintainability issues, including aggregated metrics and advanced Machine Learning (ML) models. This study benchmarks several maintainability prediction approaches, including State-of-the-Art (SotA) ML, SonarQube's Maintainability Rating, CodeScene's Code Health, and Microsoft's Maintainability Index. Our results indicate that CodeScene matches the accuracy of SotA ML and outperforms the average human expert. Importantly, unlike SotA ML, CodeScene also provides end users with actionable code smell details to remedy identified issues. Finally, caution is advised with SonarQube due to its tendency to generate many false positives. Unfortunately, our findings call into question the validity of previous studies that solely relied on SonarQube output for establishing ground truth labels. To improve reliability in future maintainability and technical debt studies, we recommend employing more accurate metrics. Moreover, reevaluating previous findings with Code Health would mitigate this revealed validity threat.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑