arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.09997cs.SEcs.AIcs.CL
收藏

Collu-Bench:一个用于预测代码中语言模型幻觉的基准

Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code

Purdue University(普渡大学)

Nan Jiang, Qi Li, Lin Tan, Tianyi Zhang

AI总结:

本文提出Collu-Bench基准,包含13,234个代码幻觉实例,用于预测LLMs在代码生成和自动化程序修复任务中的幻觉,实验表明传统机器学习和神经网络的预测准确率为22.03%至33.15%。

摘要:

尽管取得了成功,大型语言模型(LLMs)仍面临幻觉这一关键挑战,即生成看似合理但不正确的内容。尽管许多研究关注了包括图像和自然语言文本在内的多种模态中的幻觉,但对源代码中的幻觉关注较少,这会导致不正确且存在漏洞的代码,从而造成重大财务损失。为了为LLMs在代码中的幻觉研究铺平道路,我们引入了Collu-Bench,这是一个用于预测LLMs在代码生成(CG)和自动化程序修复(APR)任务中代码幻觉的基准。Collu-Bench包含从五个数据集和11个不同的LLMs(涵盖开源模型到商业模型)收集的13,234个代码幻觉实例。为了更好地理解和预测代码幻觉,Collu-Bench提供了详细的特征,例如LLMs输出的逐步对数概率、token类型以及LLMs生成代码的执行反馈,以便进行深入分析。此外,我们使用传统机器学习技术和神经网络在Collu-Bench上进行了预测幻觉的实验,准确率达到了22.03%至33.15%。我们的实验得出了关于代码幻觉模式的深刻见解,揭示了准确定位LLMs幻觉的挑战,并强调了对更复杂技术的需求。

英文摘要:

Despite their success, large language models (LLMs) face the critical challenge of hallucinations, generating plausible but incorrect content. While much research has focused on hallucinations in multiple modalities including images and natural language text, less attention has been given to hallucinations in source code, which leads to incorrect and vulnerable code that causes significant financial loss. To pave the way for research in LLMs' hallucinations in code, we introduce Collu-Bench, a benchmark for predicting code hallucinations of LLMs across code generation (CG) and automated program repair (APR) tasks. Collu-Bench includes 13,234 code hallucination instances collected from five datasets and 11 diverse LLMs, ranging from open-source models to commercial ones. To better understand and predict code hallucinations, Collu-Bench provides detailed features such as the per-step log probabilities of LLMs' output, token types, and the execution feedback of LLMs' generated code for in-depth analysis. In addition, we conduct experiments to predict hallucination on Collu-Bench, using both traditional machine learning techniques and neural networks, which achieves 22.03 -- 33.15% accuracy. Our experiments draw insightful findings of code hallucination patterns, reveal the challenge of accurately localizing LLMs' hallucinations, and highlight the need for more sophisticated techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑