arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.20197cs.SE
收藏

大型语言模型代码生成鲁棒性的初步研究

A Preliminary Study on the Robustness of Code Generation by Large Language Models

School of Software Engineering Sun Yat-sen University Zhuhai China;School of Computer Science Fudan University Shanghai China;School of Software Engineering Sun Yat-sen University;School of Computer Science Fudan University

Zike Li, Mingwei Liu, Anji Li, Kaifeng He, Yanlin Wang, Xin Peng, Zibin Zheng

AI总结:

本研究针对LLM代码生成鲁棒性评估缺失问题,基于CoderEval基准实证分析4个代码LLM,发现条件检查缺失是主要缺陷,进而提出模型无关框架RobGen,通过行级干预与标记级logit调整提升鲁棒性,实验证明其有效降低缺陷占比并提升性能。

摘要:

鲁棒性是大型语言模型可靠生成代码的关键因素,但大多数评估聚焦于正确性,却忽视了输入验证缺失、错误处理不足等核心问题。本研究首次基于CoderEval基准对LLM生成代码的鲁棒性开展实证分析。评估4个最先进的代码LLM后发现,其35.2%的输出鲁棒性低于人类编写的代码,超90%的缺陷源于条件检查缺失——其中70%发生在第一行。有趣的是,在63%需要条件语句却未生成的案例中,“if”标记仍位列前三预测,表明模型对控制流存在隐式识别。为解决这些问题,我们提出无需重新训练即可提升鲁棒性的模型无关框架RobGen。该框架结合行级干预检查器(决定是否调整每行生成的logits)与标记级条件logit调整,以促进必要控制结构生成。实验显示,RobGen将鲁棒性不足代码的占比降低10%,实现最高平均Pass@1(43.57),且仅增加33.4%的极小开销。作为轻量且适应性强的方案,RobGen有效提升了LLM生成代码在多任务中的可靠性。

英文摘要:

Robustness is a critical factor for reliable code generation by large language models, yet most evaluations focus on correctness and overlook key issues such as missing input validation and inadequate error handling. In this work, we present the first empirical study of LLM-generated code robustness using the CoderEval benchmark. Evaluating four state-of-the-art code LLMs, we find that 35.2% of their outputs are less robust than human-written code, with over 90% of deficiencies caused by missing conditional checks-70% of which occur in the first line. Interestingly, in 63% of cases where a conditional statement is needed but absent, the "if" token still ranks among the top three predictions, suggesting implicit recognition of control flow. To address these issues, we propose RobGen, a model-agnostic framework that improves robustness without retraining. RobGen combines a line-level intervention checker, which decides whether to adjust logits for each generated line, with token-level conditional logit adjustments to promote essential control structures. Experiments show that RobGen reduces the proportion of less robust code by 10%, achieves the highest average Pass@1 (43.57), and adds minimal overhead (+33.4%). As a lightweight and adaptable solution, RobGen effectively enhances the reliability of LLM-generated code across diverse tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑