De-Hallucinator：通过迭代接地缓解代码生成任务中LLM的幻觉问题
De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding
AI总结：
针对大语言模型在代码生成中易虚构不存在API的幻觉问题，提出De-Hallucinator方法，基于初始预测迭代检索适配的项目API参考加入提示，在5种LLM的两类代码任务上均显著提升生成效果。
摘要：
在公开源代码数据集上训练的大语言模型（LLM）已在代码生成任务中确立了新的最优水平。然而，这些模型大多不了解特定项目内已有的代码，导致模型无法充分利用现有API。相反，LLM经常虚构或“产生幻觉”，生成不存在的API，或是产出已有代码的变体。本文提出了De-Hallucinator，这一技术通过检索合适的API参考、并在提示词中加入越来越适配的上下文信息来迭代查询模型的新颖组合，为LLM的预测结果提供事实依据。该方法利用了如下观察：LLM的预测结果通常与期望代码相似，但无法正确引用已有的API。De-Hallucinator会自动识别与模型初始预测相关的项目特定API参考，并将这些参考添加到提示词中。与检索增强生成（RAG）不同，我们的方法利用模型的初始预测来迭代检索越来越合适的API参考。我们的评估将该方法应用于两项任务：Python中的API使用预测，以及JavaScript中的测试生成。我们表明，De-Hallucinator在5种LLM上都能持续提升生成代码的质量。具体而言，对于代码补全任务，该方法将编辑距离降低了23.3%-50.6%，将正确预测的API使用的召回率提升了23.9%-61.0%；对于测试生成任务，该方法将最初因幻觉而失败的修复测试数量提升了63.2%，最终使语句覆盖率提高了15.5%。
英文摘要：
Large language models (LLMs) trained on datasets of publicly available source code have established a new state of the art in code generation tasks. However, these models are mostly unaware of the code that exists within a specific project, preventing the models from making good use of existing APIs. Instead, LLMs often invent, or "hallucinate", non-existent APIs or produce variants of already existing code. This paper presents De-Hallucinator, a technique that grounds the predictions of an LLM through a novel combination of retrieving suitable API references and iteratively querying the model with increasingly suitable context information in the prompt. The approach exploits the observation that predictions by LLMs often resemble the desired code, but they fail to correctly refer to already existing APIs. De-Hallucinator automatically identifies project-specific API references related to the model's initial predictions and adds these references into the prompt. Unlike retrieval-augmented generation (RAG), our approach uses the initial prediction(s) by the model to iteratively retrieve increasingly suitable API references. Our evaluation applies the approach to two tasks: predicting API usages in Python and generating tests in JavaScript. We show that De-Hallucinator consistently improves the generated code across five LLMs. In particular, the approach improves the edit distance by 23.3-50.6% and the recall of correctly predicted API usages by 23.9-61.0% for code completion, and improves the number of fixed tests that initially failed because of hallucinations by 63.2%, resulting in a 15.5% increase in statement coverage for test generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
