arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.02379cs.LG
收藏

TensorGRaD:用于内存高效神经算子训练的张量梯度鲁棒分解

TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Training

University of Copenhagen(哥本哈根大学);California Institute of Technology(加州理工学院);Meta FAIR;NVIDIA AI

Sebastian Loeschcke, David Pitt, Robert Joseph George, Jiawei Zhao, Cheng Luo, Yuandong Tian, Jean Kossaifi, Anima Anandkumar

AI总结:

提出TensorGRaD方法,通过鲁棒张量分解和混合精度训练解决神经算子高分辨率训练的内存瓶颈,在保持精度的同时减少超50%内存。

摘要:

科学问题需要解析跨不同分辨率的多尺度现象,并在无限维函数空间中学习解算子。神经算子为此提供了一个强大的框架,利用张量参数化层来捕获复杂的多维关系。然而,将神经算子扩展到高分辨率问题会导致巨大的计算需求,使得工业规模模型的训练变得不可行。在本研究中,我们引入了TensorGRaD,这是一种直接解决与优化大型张量结构权重相关的内存挑战的新方法。我们的方法基于鲁棒张量分解,将梯度分解为低秩张量与稀疏张量之和,以高效捕获优化器状态(包括异常值)内的信息。此外,我们提供了TensorGRaD的混合精度训练方案,在不牺牲精度的前提下实现了进一步的内存节省。我们在Fourier Neural Operators上展示了TensorGRaD的有效性,该类模型对于求解偏微分方程(PDE)至关重要。我们为TensorGRaD提供了理论保证,证明了其相对于基于矩阵的梯度压缩方法的基本优势。我们在各种PDE任务上实证展示了巨大的改进,包括雷诺数为$10^5$的具有挑战性的湍流Navier-Stokes案例。TensorGRaD将总内存使用量减少了50%以上,同时保持甚至有时提高了精度。

英文摘要:

Scientific problems require resolving multi-scale phenomena across different resolutions and learning solution operators in infinite-dimensional function spaces. Neural operators provide a powerful framework for this, using tensor-parameterized layers to capture complex, multi-dimensional relationships. However, scaling neural operators to high-resolution problems leads to significant computational demands, making the training of industrial-scale models prohibitive. In this work, we introduce \textbf{TensorGRaD}, a novel method that directly addresses the memory challenges associated with optimizing large tensor-structured weights. Our approach, based on a \texit{robust tensor decomposition}, factorizes gradients as the sum of a low-rank tensor and a sparse one to efficiently capture information within optimizer states, including outliers. Additionally, we provide a recipe for mixed precision training of TensorGRaD, achieving further memory savings without sacrificing accuracy. We showcase the effectiveness of TensorGRaD on Fourier Neural Operators, a class of models crucial for solving partial differential equations (PDE). We provide theoretical guarantees for TensorGRaD, demonstrating its fundamental advantage over matrix-based gradient compression methods. We empirically demonstrate large improvements across various PDE tasks, including the challenging turbulent Navier-Stokes case at a Reynolds number of $10^5$. TensorGRaD reduces total memory usage by over $50\%$ while maintaining and sometimes even improving accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑