用于低记忆预处理的极端张量
Extreme Tensoring for Low-Memory Preconditioning
摘要:
最先进的模型现在使用数十亿个参数进行训练,在内存消耗方面达到了硬件极限。这就产生了对内存高效优化器的需求。为此,我们调查的限制和性能权衡的内存高效的自适应预处理梯度方法。我们提出了高维随机优化的极端张量,表明优化器需要很少的内存来受益于自适应预处理。我们的技术适用于任意模型(不一定具有张量形状的参数),并伴随着遗憾和收敛保证,这揭示了预处理器质量和表现力之间的权衡。在大规模NLP模型上,我们将优化器的内存开销降低了三个数量级,而不会降低性能。
英文摘要:
State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods. We propose extreme tensoring for high-dimensional stochastic optimization, showing that an optimizer needs very little memory to benefit from adaptive preconditioning. Our technique applies to arbitrary models (not necessarily with tensor-shaped parameters), and is accompanied by regret and convergence guarantees, which shed light on the tradeoffs between preconditioner quality and expressivity. On a large-scale NLP model, we reduce the optimizer memory overhead by three orders of magnitude, without degrading performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。