对检测和缓解代码库幻觉的静态分析方法的实证分析
An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations
AI总结：
本文通过实证分析探讨静态分析方法在检测和缓解代码库幻觉中的效果，发现其能检测部分错误和幻觉，但存在局限性。
摘要：
尽管已有大量研究，大型语言模型在生成代码时仍会幻觉，尤其是在使用库时。在需要库使用的自然语言到代码基准测试中，我们发现LLM在8.1-40%的响应中使用了不存在的库功能。一种直观的检测和缓解幻觉的方法是静态分析。本文分析了静态分析工具的潜力，包括它们能解决和无法解决的问题。我们发现静态分析工具能检测16-70%的所有错误，以及14-85%的库幻觉，性能因LLM和数据集而异。通过手动分析，我们识别出静态方法无法合理捕捉的案例，这为它们的潜力提供了上限，从48.5%到77%。总体而言，我们表明静态分析方法是解决某些形式幻觉的低成本方法，并量化了其在解决该问题上的差距。
英文摘要：
Despite extensive research, Large Language Models continue to hallucinate when generating code, particularly when using libraries. On NL-to-code benchmarks that require library use, we find that LLMs generate code that uses non-existent library features in 8.1-40% of responses. One intuitive approach for detection and mitigation of hallucinations is static analysis. In this paper, we analyse the potential of static analysis tools, both in terms of what they can solve and what they cannot. We find that static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset. Through manual analysis, we identify cases a static method could not plausibly catch, which gives an upper bound on their potential from 48.5% to 77%. Overall, we show that static analysis methods are cheap method for addressing some forms of hallucination, and we quantify how far short of solving the problem they will always be.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
