arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.08221cs.SE
收藏

冻结巨码模型中的SE事实知识--FQN及其检索研究

SE Factual Knowledge in Frozen Giant Code Model: A Study on FQN and its Retrieval

Qing Huang, Dianshu Liao, Zhenchang Xing, Zhiqiang Yuan, Qinghua Lu, Xiwei Xu, Jiaxing Lu

摘要:

预训练的巨型代码模型(PCM)开始进入开发人员的日常实践。了解什么类型的软件知识以及有多少软件知识被打包到PCM中是将PCM纳入软件工程(SE)任务并充分释放其潜力的基础。在这项工作中,我们对最先进的PCM CoPilot中的SE事实知识进行了第一次系统研究,重点是API的完全限定名称(FullyQualified Names,缩写为FKN),这是有效代码分析,搜索和重用的基础知识。在CSDN数据分布特性的驱动下,我们在Copilot上设计了一种新的轻量级上下文学习来进行CSDN推理,它不需要像传统方法那样进行代码编译,也不需要最近的CSDN调优来进行梯度更新.我们系统地实验了五个情境学习设计因素,以确定开发人员在实践中可以采用的最佳上下文学习配置。有了这个最佳的配置,我们调查的数量的例子提示和CNON数据属性对Copilot的CNON推理能力的影响。我们的研究结果证实,Copilot存储不同的CNON知识,并可以应用于CNON推理,由于其高推理精度和不依赖于代码分析。根据我们与Copilot互动的经验,我们讨论了在CNON推理任务中改善人类与CoPilot互动的各种机会。

英文摘要:

Pre-trained giant code models (PCMs) start coming into the developers' daily practices. Understanding what types of and how much software knowledge is packed into PCMs is the foundation for incorporating PCMs into software engineering (SE) tasks and fully releasing their potential. In this work, we conduct the first systematic study on the SE factual knowledge in the state-of-the-art PCM CoPilot, focusing on APIs' Fully Qualified Names (FQNs), the fundamental knowledge for effective code analysis, search and reuse. Driven by FQNs' data distribution properties, we design a novel lightweight in-context learning on Copilot for FQN inference, which does not require code compilation as traditional methods or gradient update by recent FQN prompt-tuning. We systematically experiment with five in-context-learning design factors to identify the best in-context learning configuration that developers can adopt in practice. With this best configuration, we investigate the effects of amount of example prompts and FQN data properties on Copilot's FQN inference capability. Our results confirm that Copilot stores diverse FQN knowledge and can be applied for the FQN inference due to its high inference accuracy and non-reliance on code analysis. Based on our experience interacting with Copilot, we discuss various opportunities to improve human-CoPilot interaction in the FQN inference task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑