超越功能正确性：探索LLM生成代码中的幻觉
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
AI总结：
本文探讨了LLM生成代码中的幻觉问题，通过分类和分析建立全面的幻觉分类体系，并提出轻量级的幻觉缓解方法。
摘要：
大型语言模型（LLMs）的兴起显著推动了软件工程任务中的各种应用，尤其是在代码生成领域。尽管性能表现令人印象深刻，但LLMs容易生成幻觉，这意味着LLMs可能会产生偏离用户意图、表现出内部不一致或与现实世界知识不一致的输出，这使得在广泛的应用中部署LLMs可能具有潜在风险。现有研究主要集中在自然语言生成（NLG）领域中的幻觉调查，留下了在代码生成背景下全面理解幻觉类型、原因和影响的空白。为填补这一空白，我们对LLM生成的代码进行了主题分析，以总结和分类幻觉及其原因和影响。我们的研究建立了一个全面的代码幻觉分类法，涵盖3个主要类别和12个具体类别。此外，我们系统地分析了幻觉的分布，探讨了不同LLM和基准之间的差异。此外，我们深入分析了各种幻觉的原因和影响，旨在为幻觉缓解提供有价值的见解。最后，为了在轻量级方式下提高LLM生成代码的正确性和可靠性，我们通过提示增强技术探索无训练的幻觉缓解方法。我们相信我们的发现将为未来关于代码幻觉评估和缓解的研究提供新的视角，最终为构建更有效和可靠的代码LLM铺平道路。复制包可在https://github.com/Lorien1128/code_hallucination上获得。
英文摘要：
The rise of Large Language Models (LLMs) has significantly advanced various applications on software engineering tasks, particularly in code generation. Despite the promising performance, LLMs are prone to generate hallucinations, which means LLMs might produce outputs that deviate from users' intent, exhibit internal inconsistencies, or misaligned with the real-world knowledge, making the deployment of LLMs potentially risky in a wide range of applications. Existing work mainly focuses on investigating the hallucination in the domain of Natural Language Generation (NLG), leaving a gap in comprehensively understanding the types, causes, and impacts of hallucinations in the context of code generation. To bridge the gap, we conducted a thematic analysis of the LLM-generated code to summarize and categorize the hallucinations, as well as their causes and impacts. Our study established a comprehensive taxonomy of code hallucinations, encompassing 3 primary categories and 12 specific categories. Furthermore, we systematically analyzed the distribution of hallucinations, exploring variations among different LLMs and benchmarks. Moreover, we perform an in-depth analysis on the causes and impacts of various hallucinations, aiming to provide valuable insights into hallucination mitigation. Finally, to enhance the correctness and reliability of LLM-generated code in a lightweight manner, we explore training-free hallucination mitigation approaches by prompt enhancing techniques. We believe our findings will shed light on future research about code hallucination evaluation and mitigation, ultimately paving the way for building more effective and reliable code LLMs in the future. The replication package is available at https://github.com/Lorien1128/code_hallucination
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
