CodeShell 技术报告
CodeShell Technical Report
AI总结:
本文提出 CodeShell-Base,一个 70 亿参数、8K 上下文长度的代码基础模型,融合 GQA 和 RoPE 优化架构,并使用 1000 亿高质量 GitHub 数据训练,在 Humaneval 上超越 CodeLlama,展现出强大的代码理解与生成能力。
摘要:
代码大型语言模型标志着人工智能领域的重大突破。它们专为理解和生成编程语言而设计,显著提升了代码开发工作流的效率。在本技术报告中,我们提出了 CodeShell-Base,这是一个拥有 70 亿参数、上下文长度为 8K 的基础模型,在代码理解方面展现出卓越的能力。通过将 Grouped-Query Attention 和 Rotary Positional Embedding 引入 GPT-2,CodeShell-Base 融合了 StarCoder 和 CodeLlama 的结构优势,形成了其独特的架构设计。随后,我们精心构建了全面的数据预处理流程,包括相似数据去重、基于困惑度的数据过滤以及基于模型的数据过滤。通过该流程,我们从 GitHub 中精选了 1000 亿高质量预训练数据。得益于高质量数据,CodeShell-Base 仅在 5000 亿 tokens(5 个 epochs)上训练后,便在 Humaneval 上超越了 CodeLlama。我们在包括 Python、Java 和 C++ 在内的多种语言数据集上进行了广泛实验,结果表明我们的模型在代码理解和生成方面具备强大的基础能力。
英文摘要:
Code large language models mark a pivotal breakthrough in artificial intelligence. They are specifically crafted to understand and generate programming languages, significantly boosting the efficiency of coding development workflows. In this technical report, we present CodeShell-Base, a seven billion-parameter foundation model with 8K context length, showcasing exceptional proficiency in code comprehension. By incorporating Grouped-Query Attention and Rotary Positional Embedding into GPT-2, CodeShell-Base integrates the structural merits of StarCoder and CodeLlama and forms its unique architectural design. We then carefully built a comprehensive data pre-processing process, including similar data deduplication, perplexity-based data filtering, and model-based data filtering. Through this process, We have curated 100 billion high-quality pre-training data from GitHub. Benefiting from the high-quality data, CodeShell-Base outperforms CodeLlama in Humaneval after training on just 500 billion tokens (5 epochs). We have conducted extensive experiments across multiple language datasets, including Python, Java, and C++, and the results indicate that our model possesses robust foundational capabilities in code comprehension and generation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。