arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2406.19783cs.SEcs.CL
收藏

NLPerturbator：研究代码大语言模型对自然语言变化的鲁棒性

NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations

Zhejiang University（浙江大学）；York University（约克大学）

Junkai Chen, Zhenhao Li, Xing Hu, Xin Xia

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文系统研究了代码大语言模型对现实场景中自然语言描述变化的鲁棒性，提出自动化扰动框架NLPerturbator，涵盖18类扰动，并通过六个模型的实验证明扰动提示可显著降低代码生成性能。

摘要：

大语言模型（LLMs）在基于给定自然语言描述进行代码生成方面取得了令人鼓舞的成果。它们已被集成到开源项目和商业产品中，以促进日常编码活动。提示中的自然语言描述对于LLMs理解用户需求至关重要。先前的研究发现，LLMs对提示的变化很敏感，包括那些看似不显眼的细微变化。然而，在现实场景中，自然语言描述往往存在差异（例如，不同的格式、语法和措辞）。先前关于LLMs鲁棒性的研究通常基于随机扰动，而这种扰动在现实中可能并不会实际发生。在本文中，我们开展了一项全面的研究，以调查代码LLMs在现实场景中对自然语言描述变化的鲁棒性如何。基于文献综述和对从业者的在线调查，我们总结了18类自然语言扰动以及3类共现类别的组合。我们提出了一个自动化框架NLPerturbator，它可以在给定一组提示的情况下执行每个类别的扰动。通过使用六个代码LLMs进行代码生成的一系列实验，我们发现扰动后的提示会使代码生成的性能大幅下降（例如，最多下降21.2%，平均下降4.8%至6.1%）。我们的研究强调了增强LLMs对提示中现实世界变化的鲁棒性的重要性，以及精心构建提示的必要性。

英文摘要：

Large language models (LLMs) achieve promising results in code generation based on a given natural language description. They have been integrated into open-source projects and commercial products to facilitate daily coding activities. The natural language description in the prompt is crucial for LLMs to comprehend users' requirements. Prior studies uncover that LLMs are sensitive to the changes in the prompts, including slight changes that look inconspicuous. However, the natural language descriptions often vary in real-world scenarios (e.g., different formats, grammar, and wording). Prior studies on the robustness of LLMs are often based on random perturbations and such perturbations may not actually happen. In this paper, we conduct a comprehensive study to investigate how are code LLMs robust to variations of natural language description in real-world scenarios. We summarize 18 categories of perturbations of natural language and 3 combinations of co-occurred categories based on our literature review and an online survey with practitioners. We propose an automated framework, NLPerturbator, which can perform perturbations of each category given a set of prompts. Through a series of experiments on code generation using six code LLMs, we find that the perturbed prompts can decrease the performance of code generation by a considerable margin (e.g., up to 21.2%, and 4.8% to 6.1% on average). Our study highlights the importance of enhancing the robustness of LLMs to real-world variations in the prompts, as well as the essentiality of attentively constructing the prompts.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑