Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgen… 展开作者
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries
The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40\% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.
Hugging Face;ServiceNow Research(ServiceNow研究院);Nvidia(英伟达);Northeastern University(东北大学);Independent(独立研究者);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Johns Hopkins University(约翰斯·霍普金斯大学);Leipzig University(莱比锡大学);Roblox(罗布乐思);Sea AI Lab(Sea AI实验室);Technion – Israel Institute of Technology(以色列理工学院);Monash University(莫纳什大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织Data61);University of British Columbia(不列颠哥伦比亚大学);MIT(麻省理工学院);Technical University of Munich(慕尼黑工业大学);IBM Research(IBM研究院);University of Notre Dame(圣母大学);Princeton University(普林斯顿大学);Wellesley College(韦尔斯利学院);University College London(伦敦大学学院);Kaggle;UC San Diego(加州大学圣迭戈分校);Cornell University(康奈尔大学);UC Berkeley(加州大学伯克利分校);Mazzuma;Contextual AI;Cohere;Salesforce(赛富时);Technical University of Darmstadt(达姆施塔特工业大学);Baidu(百度);Yale University(耶鲁大学)
EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization
The University of Hong Kong(香港大学);University of Edinburgh(爱丁堡大学);University College Dublin(都柏林大学学院);Shanghai AI Laboratory(上海人工智能实验室);University of Cambridge(剑桥大学)
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Babelscape(巴别景观公司);LAION(LAION组织);TU Darmstadt(达姆施塔特工业大学);UNC Chapel-Hill(北卡罗来纳大学教堂山分校);Detomo Inc.(Detomo公司);CSIRO’s Data61(联邦科学与工业研究组织Data61);Monash University(莫纳什大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for AI(艾伦人工智能研究所);DeepTensor AB(DeepTensor AB公司);WASP Media & Language(WASP媒体与语言机构);Umeå University(于默奥大学);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);TU Wien(维也纳技术大学);Institute of Science Tokyo(东京科学大学);Hudson Labs(哈德逊实验室);Luleå University of Technology(吕勒奥理工大学);ETH Zurich(苏黎世联邦理工学院);RunPod(RunPod公司);OpenChat(OpenChat机构);RIKEN CCS(理化学研究所计算科学中心);Chattermill AI(Chattermill人工智能公司);ASC27(ASC27机构);University of Massachuset(马萨诸塞大学)
Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach
National Engineering Research Center for Big Data Technology and System(国家大数据技术与系统工程研究中心);Huazhong University of Science and Technology(华中科技大学);Chongqing University(重庆大学);University of Leigh(利哈伊大学)
CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair
A Prompt Learning Framework for Source Code Summarization
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Faculty of Data Science, City University of Macau(澳门城市大学数据科学学院)