CoLA:利用组合结构实现自动且高效的数值线性代数
CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra
AI总结:
本文提出 CoLA 框架,通过线性算子抽象与组合分派规则自动构建高效数值线性代数算法,支持自动微分、低精度计算和 GPU 加速,并在偏微分方程、高斯过程等应用中验证了有效性。
摘要:
机器学习和科学的许多领域都涉及大规模线性代数问题,例如特征分解、求解线性方程组、计算矩阵指数以及迹估计。所涉及的矩阵通常具有 Kronecker 积、卷积、块对角、求和或乘积结构。在本文中,我们提出了一个简单但通用的机器学习大规模线性代数问题框架,命名为 CoLA(Compositional Linear Algebra,组合线性代数)。通过将线性算子抽象与组合分派规则相结合,CoLA 自动构建内存和运行时高效的数值算法。此外,CoLA 在 JAX 和 PyTorch 中都提供了内存高效的自动微分、低精度计算和 GPU 加速,同时还通过多重分派兼容下游包中的新对象、操作和规则。CoLA 可以加速许多代数运算,同时使矩阵结构和算法的原型设计变得容易,为几乎任何需要线性代数的计算工作提供了一个有吸引力的即插即用工具。我们展示了它在广泛应用中的有效性,包括偏微分方程、高斯过程、等变模型构建和无监督学习。
英文摘要:
Many areas of machine learning and science involve large linear algebra problems, such as eigendecompositions, solving linear systems, computing matrix exponentials, and trace estimation. The matrices involved often have Kronecker, convolutional, block diagonal, sum, or product structure. In this paper, we propose a simple but general framework for large-scale linear algebra problems in machine learning, named CoLA (Compositional Linear Algebra). By combining a linear operator abstraction with compositional dispatch rules, CoLA automatically constructs memory and runtime efficient numerical algorithms. Moreover, CoLA provides memory efficient automatic differentiation, low precision computation, and GPU acceleration in both JAX and PyTorch, while also accommodating new objects, operations, and rules in downstream packages via multiple dispatch. CoLA can accelerate many algebraic operations, while making it easy to prototype matrix structures and algorithms, providing an appealing drop-in tool for virtually any computational effort that requires linear algebra. We showcase its efficacy across a broad range of applications, including partial differential equations, Gaussian processes, equivariant model construction, and unsupervised learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。