CodeMirage：大型语言模型生成代码中的幻觉
CodeMirage: Hallucinations in Code Generated by Large Language Models
AI总结：
本文首次系统研究大型语言模型生成代码中的幻觉问题，提出代码幻觉定义、分类法及首个基准数据集CodeMirage，并实验评估多种LLM的检测性能，GPT-4表现最佳。
摘要：
大型语言模型（LLMs）在程序生成和无代码自动化方面展现出了巨大的潜力。然而，LLMs容易产生幻觉，即生成听起来合理但不正确的内容。尽管近期针对文本生成的LLM幻觉研究激增，但类似的幻觉现象也可能发生在代码生成中。有时生成的代码可能存在语法或逻辑错误，以及更高级的问题，如安全漏洞、内存泄漏等。鉴于LLMs在代码生成和一般开发中为提高效率而被广泛采用，研究代码生成中的幻觉问题变得至关重要。据我们所知，这是首次尝试研究LLMs生成代码中的幻觉。我们首先介绍了代码幻觉的定义以及代码幻觉类型的全面分类法。我们提出了第一个用于代码幻觉的基准数据集CodeMirage。该基准包含1,137个由GPT-3.5生成的、针对来自两个基础数据集（HumanEval和MBPP）的Python编程问题的幻觉代码片段。接着，我们提出了代码幻觉检测的方法论，并使用一次性提示（one-shot prompt）对开源LLMs（如CodeLLaMA）以及OpenAI的GPT-3.5和GPT-4模型进行了实验。我们发现GPT-4在HumanEval数据集上表现最佳，并在MBPP数据集上给出了与微调的CodeBERT基线相当的结果。最后，我们讨论了针对代码幻觉的各种缓解策略，并总结了我们的工作。
英文摘要：
Large Language Models (LLMs) have shown promising potentials in program generation and no-code automation. However, LLMs are prone to generate hallucinations, i.e., they generate text which sounds plausible but is incorrect. Although there has been a recent surge in research on LLM hallucinations for text generation, similar hallucination phenomenon can happen in code generation. Sometimes the generated code can have syntactical or logical errors as well as more advanced issues like security vulnerabilities, memory leaks, etc. Given the wide adaptation of LLMs to enhance efficiency in code generation and development in general, it becomes imperative to investigate hallucinations in code generation. To the best of our knowledge, this is the first attempt at studying hallucinations in the code generated by LLMs. We start by introducing the code hallucination definition and a comprehensive taxonomy of code hallucination types. We propose the first benchmark CodeMirage dataset for code hallucinations. The benchmark contains 1,137 GPT-3.5 generated hallucinated code snippets for Python programming problems from two base datasets - HumanEval and MBPP. We then propose the methodology for code hallucination detection and experiment with open source LLMs such as CodeLLaMA as well as OpenAI's GPT-3.5 and GPT-4 models using one-shot prompt. We find that GPT-4 performs the best on HumanEval dataset and gives comparable results to the fine-tuned CodeBERT baseline on MBPP dataset. Towards the end, we discuss various mitigation strategies for code hallucinations and conclude our work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
