arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.23866cs.SE
收藏

评估与改进基于ChatGPT的缩写扩展方法

Evaluating and Improving ChatGPT-Based Expansion of Abbreviations

Yanjie Jiang, Hui Liu, Lu Zhang

AI总结:

本研究首次实证探究大语言模型在源代码缩写扩展任务中的表现,针对ChatGPT准确率不足的问题,通过优化上下文选择、迭代识别标记缩写及后置条件检查,使其效果达到当前最优水平且无需复杂源码解析。

摘要:

源代码标识符通常包含缩写,这类缩写可能会降低源代码的可读性,进而阻碍软件应用的维护。为此,人们亟需精准、自动化的源代码缩写扩展方法,且缩写扩展问题已得到广泛研究。但据我们所知,现有方法大多基于启发式规则,尚无方法采用深度学习技术,更不用说最先进的大语言模型(LLM)了。LLM已在各类软件工程任务中展现出顶尖性能,因此具备自动扩展缩写的潜力。\n为此,本文首次开展了基于LLM的缩写扩展实证研究。我们在一个公开基准上的评估结果表明,ChatGPT的准确率远低于当前最优方法,其精确率和召回率分别降低了28.2%和27.8%。我们手动分析了失败案例,发现失败的根本原因有两点:1)缺乏上下文;2)无法识别缩写。针对第一个原因,我们研究了多种上下文的作用,发现周边源代码是最佳选择。针对第二个原因,我们设计了一种迭代方法,可在提示词中识别并明确标记遗漏的缩写。最后,我们提出了后置条件检查机制,以排除违反常识的错误扩展结果。所有这些措施相结合,使基于ChatGPT的缩写扩展效果可与当前最优方法相媲美,同时避免了现有最优方法必不可少的、成本高昂的源代码解析与深度分析。

英文摘要:

Source code identifiers often contain abbreviations. Such abbreviations may reduce the readability of the source code, which in turn hinders the maintenance of the software applications. To this end, accurate and automated approaches to expanding abbreviations in source code are desirable and abbreviation expansion has been intensively investigated. However, to the best of our knowledge, most existing approaches are heuristics, and none of them has even employed deep learning techniques, let alone the most advanced large language models (LLMs). LLMs have demonstrated cutting-edge performance in various software engineering tasks, and thus it has the potential to expand abbreviation automatically. To this end, in this paper, we present the first empirical study on LLM-based abbreviation expansion. Our evaluation results on a public benchmark suggest that ChatGPT is substantially less accurate than the state-of-the-art approach, reducing precision and recall by 28.2\% and 27.8\%, respectively. We manually analyzed the failed cases, and discovered the root causes for the failures: 1) Lack of contexts and 2) Inability to recognize abbreviations. In response to the first cause, we investigated the effect of various contexts and found surrounding source code is the best selection. In response to the second cause, we designed an iterative approach that identifies and explicitly marks missed abbreviations in prompts. Finally, we proposed a post-condition checking to exclude incorrect expansions that violate commonsense. All such measures together make ChatGPT-based abbreviation expansion comparable to the state of the art while avoiding expensive source code parsing and deep analysis that are indispensable for state-of-the-art approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑