通过测试执行识别LLM生成的代码注释中的不准确描述
Identifying Inaccurate Descriptions in LLM-generated Code Comments via Test Execution
AI总结:
针对LLM生成的代码注释事实准确性难评估、现有一致性检测技术无效的问题,提出基于LLM生成测试用例并执行的文档测试方法,验证Java注释时与准确性有显著统计关联,为该问题提供了新解决路径。
摘要:
软件注释对于人类理解软件至关重要,因此已有众多注释生成技术被提出。然而我们发现,针对生成注释的事实准确性的系统性评估十分罕见,目前仅给出了主观的准确性标签。我们对三个大语言模型(LLM)生成的注释展开评估,发现即便是性能最优的LLM,其生成的注释中也约有五分之一包含可证实的不准确表述。尽管代码-注释一致性检测技术看似应当能够检测出不准确的注释,但我们开展的实验表明,这类技术与注释准确性之间不存在统计学上的显著关联,凸显了该问题的巨大难度。为解决这一问题,我们提出文档测试的概念:利用LLM基于文档生成测试用例,运行这些测试并观察其是否通过,以此验证文档的准确性。此外,我们实现了该概念以验证Java注释。实验表明,我们的方法与注释准确性之间存在稳固的统计学关联,在先前技术失效的问题上取得了进展。定性评估也显示出我们的方法在赢得开发者信任方面的潜力,同时也指明了当前实现的局限性。
英文摘要:
Software comments are critical for human understanding of software, and as such many comment generation techniques have been proposed. However, we find that a systematic evaluation of the factual accuracy of generated comments is rare; only subjective accuracy labels have been given. Evaluating comments generated by three Large Language Models (LLMs), we find that even for the best-performing LLM, roughly a fifth of its comments contained demonstrably inaccurate statements. While it seems code-comment consistency detection techniques should be able to detect inaccurate comments, we perform experiments demonstrating they have no statistically significant relationship with comment accuracy, underscoring the substantial difficulty of this problem. To tackle this, we propose the concept of document testing, in which a document is verified by using an LLM to generate tests based on the document, running those tests, and observing whether they pass or fail. Furthermore, we implement our concept to verify Java comments. Experiments demonstrate that our approach has a robust statistical relationship with comment accuracy, making headway into a problem where prior techniques failed. Qualitative evaluation also reveals the promise of our approach in gaining developer trust, while highlighting the limitations of our current implementation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。