关于利用API文档缓解代码大语言模型幻觉的研究
On Mitigating Code LLM Hallucinations with API Documentation
AI总结：
本研究提出CloudAPIBench基准，并引入智能触发文档增强生成（DAG）的方法，以缓解代码大语言模型在低频API上的幻觉问题，同时避免对高频API的负面影响，最终提升API调用的可靠性。
摘要：
在本研究中，我们解决了各种软件工程环境中API幻觉的问题。我们引入了CloudAPIBench，这是一个旨在衡量API幻觉发生情况的新基准。CloudAPIBench还提供了API在公共领域出现频率的标注，使我们能够研究不同频率水平下的API幻觉。我们的研究结果显示，代码大语言模型在处理低频API时存在困难：例如，GPT-4o在低频API调用中仅有38.58%的有效率。我们证明，文档增强生成（DAG）显著提升了低频API的性能（使用DAG后提升至47.94%），但在使用次优检索器时对高频API产生负面影响（绝对下降39.02%）。为了缓解这一问题，我们提出智能触发DAG，即通过检查API索引或利用代码大语言模型的置信度分数，仅在需要时进行检索。我们证明，我们提出的方法增强了低频和高频API性能之间的平衡，从而实现了更可靠的API调用（GPT-4o在CloudAPIBench上绝对提升8.20%）。
英文摘要：
In this study, we address the issue of API hallucinations in various software engineering contexts. We introduce CloudAPIBench, a new benchmark designed to measure API hallucination occurrences. CloudAPIBench also provides annotations for frequencies of API occurrences in the public domain, allowing us to study API hallucinations at various frequency levels. Our findings reveal that Code LLMs struggle with low frequency APIs: for e.g., GPT-4o achieves only 38.58% valid low frequency API invocations. We demonstrate that Documentation Augmented Generation (DAG) significantly improves performance for low frequency APIs (increase to 47.94% with DAG) but negatively impacts high frequency APIs when using sub-optimal retrievers (a 39.02% absolute drop). To mitigate this, we propose to intelligently trigger DAG where we check against an API index or leverage Code LLMs' confidence scores to retrieve only when needed. We demonstrate that our proposed methods enhance the balance between low and high frequency API performance, resulting in more reliable API invocations (8.20% absolute improvement on CloudAPIBench for GPT-4o).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
