Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry… 展开作者
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, Harm de Vries
大语言模型(LLM)在人工智能(AI)领域发挥着越来越重要的作用-不仅用于自然语言处理,而且用于代码理解和生成。为了促进对LLM代码的开放和负责任的研究,我们引入了The Stack,这是一个3.1 TB的数据集,由30种程序设计语言的许可源代码组成。我们描述了如何收集完整的数据集,构建一个许可的子集,提出一个数据治理计划,讨论限制,并通过在不同的Python子集上训练350 M参数解码器,在text 2code基准测试中显示出有希望的结果。我们发现,(1)接近重复数据删除显著提高了所有实验的性能,(2)仅使用许可数据就可以匹配先前报告的HumanEval和MBPP性能。我们在https://hf.co/BigCode上提供了数据集,为开发人员提供了一个名为“Am I in The Stack”(https://hf.co/spaces/bigcode/in-the-stack)的工具来搜索The Stack以获得他们的代码副本,并提供了一个按照https://www.bigcode-project.org/docs/about/the-stack/上的说明从数据集中删除代码的过程。
英文摘要:
Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation. To stimulate open and responsible research on LLMs for code, we introduce The Stack, a 3.1 TB dataset consisting of permissively licensed source code in 30 programming languages. We describe how we collect the full dataset, construct a permissively licensed subset, present a data governance plan, discuss limitations, and show promising results on text2code benchmarks by training 350M-parameter decoders on different Python subsets. We find that (1) near-deduplicating the data significantly boosts performance across all experiments, and (2) it is possible to match previously reported HumanEval and MBPP performance using only permissively licensed data. We make the dataset available at https://hf.co/BigCode, provide a tool called "Am I in The Stack" (https://hf.co/spaces/bigcode/in-the-stack) for developers to search The Stack for copies of their code, and provide a process for code to be removed from the dataset by following the instructions at https://www.bigcode-project.org/docs/about/the-stack/.
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Northeastern University(东北大学);University of California, Berkeley(加利福尼亚大学伯克利分校);ServiceNow Research(ServiceNow研究院);Hugging Face;Roblox;Cursor AI
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Institute of Science Tokyo, Department of Computer Science(东京科学研究所,计算机科学系);National Institute of Advanced Industrial Science and Technology(国家先进工业科学与技术研究院);Institute of Science Tokyo, Institute of Integrated Research, Supercomputing Research Center(东京科学研究所,整合研究所,超级计算研究中心)
MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
Ontocord;Northeastern University(东北大学);LAION;Open-Ψ (Open-Sci) Collective(Open-Ψ(开放科学)联盟);Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)(于利希研究中心(FZJ)于利希超级计算中心(JSC));Carnegie Mellon University(卡内基梅隆大学);Salesforce(赛富时);Detomo Inc.(Detomo公司);Institute of Science Tokyo(东京科学研究所);École Polytechnique, IP Paris(巴黎理工学院);ELLIS Institute Tuebingen(图宾根ELLIS研究所);University of Freiburg(弗莱堡大学);NASK;Montreal Institute for Learning Algorithms, Université de Montréal(蒙特利尔大学蒙特利尔学习算法研究所);Lund University(隆德大学);DeepTensor AB