arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18800cs.SEcs.AIcs.LG
收藏

在代码生成上的大语言模型操作鲁棒性

Operational Robustness of LLMs on Code Generation

School of Engineering, Computing and Mathematics, Oxford Brookes University(工程、计算与数学学院,奥克斯伯里大学)

Debalina Ghosh Paul, Hong Zhu, Ian Bayley

AI总结:

本文提出了一种评估大语言模型在代码生成任务中鲁棒性的方法,通过分析自然语言描述的最小变化来评估模型性能,并对四种先进模型进行了实验验证。

摘要:

在软件开发中,使用大语言模型(LLMs)生成程序代码已成为一种常见实践。评估LLMs在此用途上的鲁棒性是有必要的。本文特别关注LLMs对编码任务描述变化的敏感性。然而,现有的评估鲁棒性的技术不适合代码生成,因为自然语言描述的输入数据空间是离散的。为了解决这个问题,我们提出了一种鲁棒性评估方法,称为场景领域分析,旨在找到使LLMs产生错误输出的自然语言描述的预期最小变化。我们已正式证明了该方法的理论性质,并进行了广泛的实验,以评估四个最先进的LLMs:Gemini-pro、Codex、Llamma2和Falcon 7B的鲁棒性,并发现能够以信心对它们进行从最好到最差的排名。此外,我们还研究了鲁棒性在不同场景中的变化,包括编码任务主题的变化和其样本解决方案的复杂性变化,并发现更复杂的任务和更高级的主题(如多线程和数据结构)的鲁棒性更低。

英文摘要:

It is now common practice in software development for large language models (LLMs) to be used to generate program code. It is desirable to evaluate the robustness of LLMs for this usage. This paper is concerned in particular with how sensitive LLMs are to variations in descriptions of the coding tasks. However, existing techniques for evaluating this robustness are unsuitable for code generation because the input data space of natural language descriptions is discrete. To address this problem, we propose a robustness evaluation method called scenario domain analysis, which aims to find the expected minimal change in the natural language descriptions of coding tasks that would cause the LLMs to produce incorrect outputs. We have formally proved the theoretical properties of the method and also conducted extensive experiments to evaluate the robustness of four state-of-the-art art LLMs: Gemini-pro, Codex, Llamma2 and Falcon 7B, and have found that we are able to rank these with confidence from best to worst. Moreover, we have also studied how robustness varies in different scenarios, including the variations with the topic of the coding task and with the complexity of its sample solution, and found that robustness is lower for more complex tasks and also lower for more advanced topics, such as multi-threading and data structures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑