arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.03136cs.SE
收藏

代码智能的自动化提示生成:一项实证研究及在微信中的实践

Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat

The Chinese University of Hong Kong(香港中文大学);WeChat Group, Tencent Inc.(腾讯公司微信事业群)

Kexing Ji, Shiyun Fu, Cuiyun Gao, Yujia Chen, Zezhou Yang, Chaozheng Wang, Yuetang Deng

AI总结:

针对代码智能中手动提示设计耗时且依赖特定模型的问题,本文实证研究了指令生成(IG)和多步推理(MSR),并提出一种结合两者最佳方法的新型自动化提示生成(APG)方法,在多个代码任务及微信工业数据集上均取得显著性能提升。

摘要:

大型代码模型(LCMs)在代码智能领域展现出潜力,但其有效性很大程度上受提示质量的影响。当前的提示设计大多是手动的,这不仅耗时,而且高度依赖于特定的LCMs和任务。尽管自然语言处理(NLP)中已存在自动化提示生成(APG),但其在代码智能领域仍缺乏充分探索。这形成了一个空白,因为对于面临多样化任务和黑盒LCMs的开发者而言,实现提示流程的自动化至关重要。为了缓解这一问题,我们对APG的两个重要部分进行了实证研究:指令生成(IG)和多步推理(MSR)。IG提供与任务相关的描述以指导LCMs,而MSR则引导它们在得出最终答案前产生逻辑步骤。我们在四个开源LCMs和三个代码智能任务上评估了各部分广泛使用的APG方法:代码翻译(PL-PL)、代码摘要(PL-NL)和API推荐(NL-PL)。实验结果表明,与基础提示相比,IG和MSR均显著提升了性能。基于这些结果,我们提出了一种结合这两部分最佳方法的新型APG方法。实验显示,与基础提示相比,我们的方法在CodeBLEU(代码翻译)上平均提升了28.38%,在ROUGE-L(代码摘要)上平均提升了58.11%,在SuccessRate@1(API推荐)上平均提升了84.53%。为了验证其在工业场景中的有效性,我们在专有数据集WeChat-Bench上评估了该方法,在API推荐任务上实现了平均148.89%的MRR提升。

英文摘要:

Large Code Models (LCMs) show potential in code intelligence, but their effectiveness is greatly influenced by prompt quality. Current prompt design is mostly manual, which is time-consuming and highly dependent on specific LCMs and tasks. While automated prompt generation (APG) exists in NLP, it is underexplored for code intelligence. This creates a gap, as automating the prompt process is essential for developers facing diverse tasks and black-box LCMs. To mitigate this, we empirically investigate two important parts of APG: Instruction Generation (IG) and Multi-Step Reasoning (MSR). IG provides a task-related description to instruct LCMs, while MSR guides them to produce logical steps before the final answer. We evaluate widely-used APG methods for each part on four open-source LCMs and three code intelligence tasks: code translation (PL-PL), code summarization (PL-NL), and API recommendation (NL-PL).Experimental results indicate that both IG and MSR dramatically enhance performance compared to basic prompts. Based on these results, we propose a novel APG approach combining the best methods of the two parts. Experiments show our approach achieves average improvements of 28.38% in CodeBLEU (code translation), 58.11% in ROUGE-L (code summarization), and 84.53% in SuccessRate@1 (API recommendation) over basic prompts. To validate its effectiveness in an industrial scenario, we evaluate our approach on WeChat-Bench, a proprietary dataset, achieving an average MRR improvement of 148.89% for API recommendation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑