立场:曲率矩阵应通过线性算子实现民主化
Position: Curvature Matrices Should Be Democratized via Linear Operators
AI总结:
本文主张通过线性算子接口处理曲率矩阵,开发了 $\textit{curvlinops}$ 库,提供通用、可扩展且易用的抽象,简化复杂计算并支持大规模神经网络应用。
摘要:
结构化大型矩阵在机器学习中很普遍。一类特别重要的是曲率矩阵,如 Hessian,它们对理解神经网络(NNs)的损失景观至关重要,并支持二阶优化、不确定性量化、模型剪枝、数据归因等。然而,由于自动微分的复杂性,以及曲率代理的多样性与结构假设(如稀疏性和 Kronecker 分解),曲率计算颇具挑战。在本立场论文中,我们主张线性算子——一种执行矩阵-向量乘积的接口——提供了一种通用、可扩展且易于使用的抽象来处理曲率矩阵。为支持此立场,我们开发了 $\textit{curvlinops}$,该库通过统一的线性算子接口提供曲率矩阵。我们用 $\textit{curvlinops}$ 展示了该接口如何隐藏复杂性、简化应用、实现可扩展性、与其他库互操作,并扩展至大型 NNs。
英文摘要:
Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed $\textit{curvlinops}$, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with $\textit{curvlinops}$ how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。