arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.15747cs.SEcs.AI
收藏

CodeShell 技术报告

CodeShell Technical Report

National Engineering Research Center for Software Engineering, Peking University(北京大学国家软件工程研究中心)

Rui Xie, Zhengran Zeng, Zhuohao Yu, Chang Gao, Shikun Zhang, Wei Ye

AI总结:

本文提出 CodeShell-Base,一个 70 亿参数、8K 上下文长度的代码基础模型,融合 GQA 和 RoPE 优化架构,并使用 1000 亿高质量 GitHub 数据训练,在 Humaneval 上超越 CodeLlama,展现出强大的代码理解与生成能力。

摘要:

代码大型语言模型标志着人工智能领域的重大突破。它们专为理解和生成编程语言而设计,显著提升了代码开发工作流的效率。在本技术报告中,我们提出了 CodeShell-Base,这是一个拥有 70 亿参数、上下文长度为 8K 的基础模型,在代码理解方面展现出卓越的能力。通过将 Grouped-Query Attention 和 Rotary Positional Embedding 引入 GPT-2,CodeShell-Base 融合了 StarCoder 和 CodeLlama 的结构优势,形成了其独特的架构设计。随后,我们精心构建了全面的数据预处理流程,包括相似数据去重、基于困惑度的数据过滤以及基于模型的数据过滤。通过该流程,我们从 GitHub 中精选了 1000 亿高质量预训练数据。得益于高质量数据,CodeShell-Base 仅在 5000 亿 tokens(5 个 epochs)上训练后,便在 Humaneval 上超越了 CodeLlama。我们在包括 Python、Java 和 C++ 在内的多种语言数据集上进行了广泛实验,结果表明我们的模型在代码理解和生成方面具备强大的基础能力。

英文摘要:

Code large language models mark a pivotal breakthrough in artificial intelligence. They are specifically crafted to understand and generate programming languages, significantly boosting the efficiency of coding development workflows. In this technical report, we present CodeShell-Base, a seven billion-parameter foundation model with 8K context length, showcasing exceptional proficiency in code comprehension. By incorporating Grouped-Query Attention and Rotary Positional Embedding into GPT-2, CodeShell-Base integrates the structural merits of StarCoder and CodeLlama and forms its unique architectural design. We then carefully built a comprehensive data pre-processing process, including similar data deduplication, perplexity-based data filtering, and model-based data filtering. Through this process, We have curated 100 billion high-quality pre-training data from GitHub. Benefiting from the high-quality data, CodeShell-Base outperforms CodeLlama in Humaneval after training on just 500 billion tokens (5 epochs). We have conducted extensive experiments across multiple language datasets, including Python, Java, and C++, and the results indicate that our model possesses robust foundational capabilities in code comprehension and generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑