arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.06161cs.CLcs.AIcs.PLcs.SE
收藏

StarCoder:愿源码与你同在!

StarCoder: may the source be with you!

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgen… 展开作者

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries

AI总结:

BigCode社区推出15.5B参数的StarCoder系列代码大模型,经大规模数据训练与微调,性能优于多款开源及商业模型,同时配套安全发布机制并以商用友好许可开源。

摘要:

致力于负责任开发代码大语言模型(Code LLMs)的开放科学协作社区BigCode推出了StarCoder与StarCoderBase:这是两款参数量为15.5B的模型,上下文长度达8K,具备代码补全能力,且借助多查询注意力实现了快速大批次推理。StarCoderBase基于源自The Stack的1万亿token训练而成,The Stack是一个收录大量获宽松许可的GitHub仓库的集合,配套有检查工具与退出流程。我们在35B Python token上对StarCoderBase进行微调,由此得到StarCoder。我们开展了迄今为止最全面的Code LLM评估,结果显示StarCoderBase的表现优于所有支持多编程语言的开源Code LLM,且与OpenAI的code-cushman-001模型表现相当或更优。此外,StarCoder的表现优于所有在Python上微调的模型,经提示后在HumanEval上的pass@1指标可达40%,同时仍能保持其在其他编程语言上的性能。我们为安全发布开放访问模型采取了多项重要举措,包括改进的个人身份信息(PII)删除流水线与一款全新的归属追踪工具,并依据更具商业可行性版本的开放负责任AI模型许可,公开发布StarCoder模型。

英文摘要:

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40\% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑