Adapprox:基于随机低秩矩阵的 Adam 优化自适应近似
Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices
AI总结:
针对 Adam 优化器的内存消耗问题,提出采用随机低秩矩阵近似二阶矩的 Adapprox 方法,通过自适应秩选择和余弦相似度引导策略平衡精度与内存,在 GPT-2 训练中显著节省内存并提升收敛速度与下游任务表现。
摘要:
随着深度学习模型规模呈指数级增长,Adam 等优化器因需存储一阶矩和二阶矩数据而面临显著的内存消耗挑战。现有的内存高效方法(如 Adafactor 和 CAME)通常因其矩阵分解技术而牺牲精度。针对此问题,我们提出了 Adapprox,这是一种新颖的方法,采用随机低秩矩阵近似来更有效且准确地近似 Adam 的二阶矩。Adapprox 具备自适应秩选择机制,能精细平衡精度与内存效率,并包含可选的余弦相似度引导策略以增强稳定性并加速收敛。在 GPT-2 训练及下游任务中,Adapprox 超越了 AdamW,在启用一阶矩的情况下,针对 117M 和 345M 模型分别实现了 34.5% 至 49.9% 和 33.8% 至 49.9% 的内存节省,在未启用一阶矩时进一步提升了节省幅度。此外,相较于同类方法,它还加快了收敛速度并改善了下游任务表现。
英文摘要:
As deep learning models exponentially increase in size, optimizers such as Adam encounter significant memory consumption challenges due to the storage of first and second moment data. Current memory-efficient methods like Adafactor and CAME often compromise accuracy with their matrix factorization techniques. Addressing this, we introduce Adapprox, a novel approach that employs randomized low-rank matrix approximation for a more effective and accurate approximation of Adam's second moment. Adapprox features an adaptive rank selection mechanism, finely balancing accuracy and memory efficiency, and includes an optional cosine similarity guidance strategy to enhance stability and expedite convergence. In GPT-2 training and downstream tasks, Adapprox surpasses AdamW by achieving 34.5% to 49.9% and 33.8% to 49.9% memory savings for the 117M and 345M models, respectively, with the first moment enabled, and further increases these savings without the first moment. Besides, it enhances convergence speed and improves downstream task performance relative to its counterparts.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。