arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.02524cs.PLcs.AI
收藏

Meta 大语言模型编译器:编译器优化的基础模型

Meta Large Language Model Compiler: Foundation Models of Compiler Optimization

Meta AI

Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, Hugh Leather

AI总结:

本文提出 Meta LLM Compiler,基于 Code Llama 在 5460 亿 token 的 LLVM-IR 与汇编代码上训练,用于代码大小优化和汇编反汇编,分别达到自动调优潜力的 77% 和 45% 往返率。

摘要:

大语言模型(LLMs)在各种软件工程和编码任务中展现了卓越的能力。然而,它们在代码和编译器优化领域的应用仍未得到充分探索。训练大语言模型是资源密集型的,需要大量的 GPU 小时数和广泛的数据收集,这可能令人望而却步。为了弥补这一空白,我们引入了 Meta 大语言模型编译器(LLM Compiler),这是一套稳健的、公开可用的预训练模型,专门为代码优化任务设计。LLM Compiler 建立在 Code Llama 的基础上,增强了对编译器中间表示(IRs)、汇编语言和优化技术的理解。该模型在包含 5460 亿个 token 的 LLVM-IR 和汇编代码的海量语料库上进行了训练,并经过了指令微调以解释编译器行为。LLM Compiler 在定制的商业许可下发布,以允许广泛复用,并提供两种规模:70 亿和 130 亿参数。我们还展示了该模型的微调版本,证明了其在优化代码大小以及将 x86_64 和 ARM 汇编反汇编回 LLVM-IR 方面的增强能力。这些微调版本实现了自动调优搜索优化潜力的 77%,以及 45% 的反汇编往返率(14% 精确匹配)。此次发布旨在为学术研究人员和工业实践者在编译器优化领域的进一步研究和开发提供一个可扩展的、具有成本效益的基础。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable capabilities across a variety of software engineering and coding tasks. However, their application in the domain of code and compiler optimization remains underexplored. Training LLMs is resource-intensive, requiring substantial GPU hours and extensive data collection, which can be prohibitive. To address this gap, we introduce Meta Large Language Model Compiler (LLM Compiler), a suite of robust, openly available, pre-trained models specifically designed for code optimization tasks. Built on the foundation of Code Llama, LLM Compiler enhances the understanding of compiler intermediate representations (IRs), assembly language, and optimization techniques. The model has been trained on a vast corpus of 546 billion tokens of LLVM-IR and assembly code and has undergone instruction fine-tuning to interpret compiler behavior. LLM Compiler is released under a bespoke commercial license to allow wide reuse and is available in two sizes: 7 billion and 13 billion parameters. We also present fine-tuned versions of the model, demonstrating its enhanced capabilities in optimizing code size and disassembling from x86_64 and ARM assembly back into LLVM-IR. These achieve 77% of the optimising potential of an autotuning search, and 45% disassembly round trip (14% exact match). This release aims to provide a scalable, cost-effective foundation for further research and development in compiler optimization by both academic researchers and industry practitioners.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑