通过位置嵌入和高效注意力机制评估长代码中的上下文长度外推
An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention
AI总结:
本文通过位置嵌入和高效注意力机制评估长代码中的上下文长度外推,旨在提升代码生成任务的泛化能力。
摘要:
大型语言模型的快速发展导致软件工程中自动化工具显著增加,能够执行代码生成、完成和翻译等任务。尽管有这些进步,其效果受到固定上下文长度的限制,限制了其在长领域特定代码序列上的泛化能力。为了解决这一挑战,我们研究了零样本、仅推理的方法,旨在改进位置编码并优化注意力机制。我们的目标是提供对当前方法的全面分析,以促进代码中的上下文长度外推,特别是在长代码完成任务中。
英文摘要:
The rapid advancement of large language models (LLMs) has led to a significant increase in automated tools in the software engineering, capable of performing various code-related tasks such as code generation, completion, and translation. Despite these advancements, its effectiveness is constrained by fixed context lengths, limiting its ability to generalize across long, domain-specific code sequences. To address this challenge, we investigate zero-shot, inference-only methods aimed at improving position encodings and optimizing attention mechanisms. Our goal is to provide a thorough analysis of current approaches that facilitate context length extrapolation in code, particularly in the context of long code completion tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。