对TensorFlow和PyTorch的线性代数意识进行基准测试
Benchmarking the Linear Algebra Awareness of TensorFlow and PyTorch
摘要：
在机器学习中普遍存在的线性代数运算形成了主要的性能瓶颈。高性能计算社区在开发特定于体系结构的优化内核方面投入了大量精力，例如BLAS和LAPACK库提供的内核，以加速线性代数运算。然而，最终用户逐渐不太可能经历直接使用所述内核的容易出错和耗时的过程;相反，诸如TensorFlow（TF）和PyTorch（PyT）之类的框架，它们促进了机器学习应用程序的开发，变得越来越流行。虽然这些框架与BLAS和LAPACK相关联，但尚不清楚它们是否利用线性代数知识来加速计算。出于这个原因，在本文中，我们开发了基准来研究TF和PyT的线性代数优化能力。我们的分析表明，一些线性代数优化仍然缺失;例如，通过应用分配律减少标量运算的数量，以及自动识别矩阵链的最佳括号化。在这项工作中，我们专注于TF和PyT中的线性代数计算;我们都暴露了性能增强的机会，以使框架的开发人员受益，并为最终用户提供如何实现性能增益的指导。
英文摘要：
Linear algebra operations, which are ubiquitous in machine learning, form major performance bottlenecks. The High-Performance Computing community invests significant effort in the development of architecture-specific optimized kernels, such as those provided by the BLAS and LAPACK libraries, to speed up linear algebra operations. However, end users are progressively less likely to go through the error prone and time-consuming process of directly using said kernels; instead, frameworks such as TensorFlow (TF) and PyTorch (PyT), which facilitate the development of machine learning applications, are becoming more and more popular. Although such frameworks link to BLAS and LAPACK, it is not clear whether or not they make use of linear algebra knowledge to speed up computations. For this reason, in this paper we develop benchmarks to investigate the linear algebra optimization capabilities of TF and PyT. Our analyses reveal that a number of linear algebra optimizations are still missing; for instance, reducing the number of scalar operations by applying the distributive law, and automatically identifying the optimal parenthesization of a matrix chain. In this work, we focus on linear algebra computations in TF and PyT; we both expose opportunities for performance enhancement to the benefit of the developers of the frameworks and provide end users with guidelines on how to achieve performance gains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
