arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.01709cs.MScs.AIcs.LGcs.PFcs.PL
收藏

代替重写机器学习的外来代码,自动合成快速响应

Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)

William S. Moses, Valentin Churavy

摘要:

将可微分编程技术和机器学习算法应用于外来程序需要开发人员在机器学习框架中重写他们的代码,或者以其他方式提供外来代码的衍生物。本文介绍了酶,一个高性能的自动微分(AD)编译器插件的LLVM编译器框架能够合成梯度的静态分析程序表示的LLVM中间表示(IR)。酶合成梯度的任何语言编写的程序,其编译器的目标LLVM IR包括C,C++,Fortran,Julia,Rust,Swift,MLIR等,从而以这些语言提供本机AD能力。与传统的源到源和操作符重载工具不同,酶在优化的IR上执行AD。在包括Microsoft ADBench在内的机器学习重点基准测试套件中,优化IR上的AD实现了比优化前IR上的AD高4.5倍的几何平均加速,使酶实现了最先进的性能。Packaging Enzyme for PyTorch和TensorFlow提供了对具有最先进性能的外来代码梯度的方便访问,使外来代码能够直接纳入现有的机器学习工作流程。

英文摘要:

Applying differentiable programming techniques and machine learning algorithms to foreign programs requires developers to either rewrite their code in a machine learning framework, or otherwise provide derivatives of the foreign code. This paper presents Enzyme, a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework capable of synthesizing gradients of statically analyzable programs expressed in the LLVM intermediate representation (IR). Enzyme synthesizes gradients for programs written in any language whose compiler targets LLVM IR including C, C++, Fortran, Julia, Rust, Swift, MLIR, etc., thereby providing native AD capabilities in these languages. Unlike traditional source-to-source and operator-overloading tools, Enzyme performs AD on optimized IR. On a machine-learning focused benchmark suite including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.5x over AD on IR before optimization allowing Enzyme to achieve state-of-the-art performance. Packaging Enzyme for PyTorch and TensorFlow provides convenient access to gradients of foreign code with state-of-the art performance, enabling foreign code to be directly incorporated into existing machine learning workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑