arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.01535cs.SE
收藏

评估、利用和缓解基于LLM的代码生成中的语法鲁棒性故障

Assessing, Exploiting, and Mitigating Syntactic Robustness Failures in LLM-Based Code Generation

University of California, Santa Barbara(加州大学圣塔芭芭拉分校)

Laboni Sarker, Mara Downing, Achintya Desai, Tevfik Bultan

AI总结:

研究评估LLM在代码生成中语法鲁棒性,发现其在包含数学公式的提示下存在缺陷,提出预处理步骤提升鲁棒性,使鲁棒性从54.05%提升至74.42%。

摘要:

大规模语言模型(LLM)的快速发展使基于LLM的代码生成成为研究重点。LLM代码生成器接收提示并生成实现提示中指定要求的代码。许多软件需求包含数学公式,以指定生成代码的预期行为。给定包含数学公式的代码生成提示,合理期望是,如果公式在不改变其语义的情况下进行语法修改,生成的修改后提示的代码应具有语义等价性。我们正式将这一概念定义为语法鲁棒性,并研究LLM作为代码生成器的语法鲁棒性。我们的实验评估表明,LLM在包含公式的代码生成提示中并不具备语法鲁棒性,特别是需要数学推理的提示。我们研究了可以进一步降低LLM语法鲁棒性的攻击策略。最后,为了缓解LLM中的语法鲁棒性故障,我们提出了一种预处理步骤,使用约简将提示中的公式转换为简化形式。我们的实验结果表明,使用我们的方法,LLM基于代码生成的语法鲁棒性显著提高,使LLM的语法鲁棒性从54.05%提升至74.42%。

英文摘要:

Rapid advances in the field of Large Language Models (LLMs) have made LLM-based code generation an important area for investigation. An LLM-based code generator takes a prompt as input and produces code that implements the requirements specified in the prompt. Many software requirements include mathematical formulas that specify the expected behavior of the code to be generated. Given a code generation prompt that contains a mathematical formula, a reasonable expectation is that, if the formula is syntactically modified without changing its semantics, the generated code for the modified prompt should be semantically equivalent. We formalize this concept as syntactic robustness and investigate the syntactic robustness of LLMs as code generators. Our experimental assessment demonstrates that LLMs are not syntactically robust for code generation prompts with formulas, especially for the ones that require mathematical reasoning. We investigate attack strategies that can further deteriorate the syntactic robustness of LLMs. Finally, to mitigate syntactic robustness failures in LLMs, we propose a pre-processing step that uses reductions to transform formulas in prompts to a simplified form. Our experimental results demonstrate that the syntactic robustness of LLM-based code generation improves significantly using our approach, improving syntactic robustness of LLMs from 54.05% to 74.42%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑