arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.15533cs.CLcs.AI
收藏

The Stack:3个TB,属于许可许可的source code

The Stack: 3 TB of permissively licensed source code

ServiceNow Research(ServiceNow研究院);Hugging Face(Hugging Face公司);ServiceNow(ServiceNow公司)
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry… 展开作者

Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, Harm de Vries

摘要:

大语言模型(LLM)在人工智能(AI)领域发挥着越来越重要的作用-不仅用于自然语言处理,而且用于代码理解和生成。为了促进对LLM代码的开放和负责任的研究,我们引入了The Stack,这是一个3.1 TB的数据集,由30种程序设计语言的许可源代码组成。我们描述了如何收集完整的数据集,构建一个许可的子集,提出一个数据治理计划,讨论限制,并通过在不同的Python子集上训练350 M参数解码器,在text 2code基准测试中显示出有希望的结果。我们发现,(1)接近重复数据删除显著提高了所有实验的性能,(2)仅使用许可数据就可以匹配先前报告的HumanEval和MBPP性能。我们在https://hf.co/BigCode上提供了数据集,为开发人员提供了一个名为“Am I in The Stack”(https://hf.co/spaces/bigcode/in-the-stack)的工具来搜索The Stack以获得他们的代码副本,并提供了一个按照https://www.bigcode-project.org/docs/about/the-stack/上的说明从数据集中删除代码的过程。

英文摘要:

Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation. To stimulate open and responsible research on LLMs for code, we introduce The Stack, a 3.1 TB dataset consisting of permissively licensed source code in 30 programming languages. We describe how we collect the full dataset, construct a permissively licensed subset, present a data governance plan, discuss limitations, and show promising results on text2code benchmarks by training 350M-parameter decoders on different Python subsets. We find that (1) near-deduplicating the data significantly boosts performance across all experiments, and (2) it is possible to match previously reported HumanEval and MBPP performance using only permissively licensed data. We make the dataset available at https://hf.co/BigCode, provide a tool called "Am I in The Stack" (https://hf.co/spaces/bigcode/in-the-stack) for developers to search The Stack for copies of their code, and provide a process for code to be removed from the dataset by following the instructions at https://www.bigcode-project.org/docs/about/the-stack/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑