arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.21049cs.CLcs.AIcs.LG
收藏

评估代码生成LLM中的长程依赖处理能力

Evaluating Long Range Dependency Handling in Code Generation LLMs

Apple(苹果公司)

Yannick Assogba, Donghao Ren

AI总结:

本文通过多步键检索任务评估代码生成LLM的长程依赖处理能力,发现函数后向引用及滑动窗口注意力机制会显著降低性能,并利用调用图信息优化提示词,将多步检索性能提升高达3倍。

摘要:

随着语言模型支持的上下文规模日益增大,评估其有效利用上下文的能力变得愈发重要。我们使用一套多步键检索任务,在长达8k token的上下文窗口中,分析了多个代码生成模型处理长程依赖的能力。这些任务的难度逐步递增,与流行的“大海捞针”测试相比,能够更细致地评估模型能力。我们发现,当函数引用在提示词中靠后定义的另一函数时,许多模型的性能显著下降(高达2倍)。我们还观察到,采用滑动窗口注意力机制的模型难以处理超出单个窗口大小的引用。我们利用调用图信息进行简单的提示词修改,将多步检索性能提升了高达3倍。我们的分析强调,长上下文性能需要超越文档内单一事实检索的更深入考量。

英文摘要:

As language models support larger and larger context sizes, evaluating their ability to make effective use of that context becomes increasingly important. We analyze the ability of several code generation models to handle long range dependencies using a suite of multi-step key retrieval tasks in context windows up to 8k tokens in length. The tasks progressively increase in difficulty and allow more nuanced evaluation of model capabilities than tests like the popular needle-in-the-haystack test. We find that performance degrades significantly for many models (up to 2x) when a function references another function that is defined later in the prompt. We also observe that models that use sliding window attention mechanisms have difficulty handling references further than the size of a single window. We perform simple prompt modifications using call graph information to improve multi-step retrieval performance up to 3x. Our analysis highlights ways that long-context performance needs deeper consideration beyond retrieval of single facts within a document.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑