arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.15228cs.SE
收藏

对指令微调模型在代码生成中系统提示效果的实证研究

An Empirical Study on the Effects of System Prompts in Instruction-Tuned Models for Code Generation

William & Mary(威廉与玛丽学院)

Zaiyu Cheng, Antonio Mastropaolo

AI总结:

本研究探讨了系统提示对代码生成模型性能的影响,发现提示具体性与模型规模、编程语言等因素相关,且不同语言对提示敏感性存在差异。

摘要:

指令微调语言模型(ILMs)已成为现代人工智能系统中的关键组成部分,在自然语言和推理任务中表现出卓越的通用性。其中最显著的应用是代码生成,其中ILMs——通常称为代码语言模型(CLMs)——将人类意图转化为可执行程序。虽然进展由模型规模和训练方法的进步推动,但一个关键方面仍未被充分探讨:系统提示对通用ILMs和专门用于代码生成的CLMs的影响。我们系统地评估了不同指令详细程度的系统提示、模型规模、提示策略和编程语言如何影响代码助手。我们的实验设置涵盖了360种配置,覆盖四种模型、五个系统提示、三种提示策略、两种语言和两种温度设置。我们发现:(1)增加系统提示的约束具体性并不总是单调提高正确性——提示效果取决于配置,并且可以根据任务要求和解码上下文帮助或阻碍;(2)对于较大的代码专门化模型,少量示例可能比零样本生成表现更差,与传统观念相反;(3)编程语言起作用,Java对系统提示的变化比Python敏感性显著更高,这表明可能需要语言特定的提示工程策略。

英文摘要:

Instruction-tuned Language Models (ILMs) have become essential components of modern AI systems, demonstrating exceptional versatility across natural language and reasoning tasks. Among their most impactful applications is code generation, where ILMs -- commonly referred to as Code Language Models (CLMs) -- translate human intent into executable programs. While progress has been driven by advances in scaling and training methodologies, one critical aspect remains underexplored: the impact of system prompts on both general-purpose ILMs and specialized CLMs for code generation. We systematically evaluate how system prompts of varying instructional detail, along with model scale, prompting strategy, and programming language, affect code assistant. Our experimental setting spans 360 configurations across four models, five system prompts, three prompting strategies, two languages, and two temperature settings. We find that (1) increasing system-prompt constraint specificity does not monotonically improve correctness -- prompt effectiveness is configuration-dependent and can help or hinder based on alignment with task requirements and decoding context; (2) for larger code-specialized models, few-shot examples can degrade performance relative to zero-shot generation, contrary to conventional wisdom; and (3) programming language matters, with Java exhibiting significantly greater sensitivity to system prompt variations than Python, suggesting language-specific prompt engineering strategies may be necessary.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑