HiRoPE:使用分层位置编码实现代码模型的长度外推
HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position
AI总结:
本文提出HiRoPE,将旋转位置嵌入按代码分层结构改进,无需额外训练即可显著扩展LLM上下文长度,并在长代码任务中验证了其稳定性能。
摘要:
解决大型语言模型在代码相关任务中上下文长度限制的问题是本文的主要关注点。现有LLM受限于其预训练上下文长度,导致在处理长而复杂的代码序列时出现性能问题。受人类程序员浏览代码方式的启发,我们提出了分层旋转位置嵌入(HiRoPE),这是一种新颖的方法,将传统的旋转位置嵌入增强为基于源代码分层结构的分层格式。HiRoPE可以轻松集成到现有LLM中,而无需额外的训练成本。我们的方法在多种LLM上进行了广泛评估,在语言建模和长代码补全等任务中表现出稳定的性能。我们还引入了一个使用真实世界代码项目的新的长代码理解任务,希望促进这一代码相关领域的进一步发展。从理论和实验上,我们发现HiRoPE还解决了位置编码中的分布外问题。我们的HiRoPE显著扩展了LLM的上下文长度能力,使得推理长度可以比训练长度呈指数级增长。
英文摘要:
Addressing the limitation of context length in large language models for code-related tasks is the primary focus of this paper. Existing LLMs are constrained by their pre-trained context lengths, leading to performance issues in handling long complex code sequences. Inspired by how human programmers navigate code, we introduce Hierarchical Rotary Position Embedding (HiRoPE), a novel approach that enhances the traditional rotary position embedding into a hierarchical format based on the hierarchical structure of source code. HiRoPE offers easy integration into existing LLMs without extra training costs. Our method is extensively evaluated with various LLMs, demonstrating stable performance in tasks such as language modeling and long code completion. We also introduce a new long code understanding task with real-world code projects, in hopes of promoting further development in this code-related field. Theoretically and experimentally, we find that HiRoPE also addresses the out-of-distribution issue in position encoding. Our HiRoPE significantly expands the context length capabilities of LLMs, enabling inference at lengths exponentially greater than the training length.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。