arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.04620cs.LGstat.ML
收藏

用于低记忆预处理的极端张量

Extreme Tensoring for Low-Memory Preconditioning

Google AI Princeton(谷歌人工智能普林斯顿);Department of Computer Science, Princeton University(普林斯顿大学计算机科学系)

Xinyi Chen, Naman Agarwal, Elad Hazan, Cyril Zhang, Yi Zhang

摘要:

最先进的模型现在使用数十亿个参数进行训练,在内存消耗方面达到了硬件极限。这就产生了对内存高效优化器的需求。为此,我们调查的限制和性能权衡的内存高效的自适应预处理梯度方法。我们提出了高维随机优化的极端张量,表明优化器需要很少的内存来受益于自适应预处理。我们的技术适用于任意模型(不一定具有张量形状的参数),并伴随着遗憾和收敛保证,这揭示了预处理器质量和表现力之间的权衡。在大规模NLP模型上,我们将优化器的内存开销降低了三个数量级,而不会降低性能。

英文摘要:

State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods. We propose extreme tensoring for high-dimensional stochastic optimization, showing that an optimizer needs very little memory to benefit from adaptive preconditioning. Our technique applies to arbitrary models (not necessarily with tensor-shaped parameters), and is accompanied by regret and convergence guarantees, which shed light on the tradeoffs between preconditioner quality and expressivity. On a large-scale NLP model, we reduce the optimizer memory overhead by three orders of magnitude, without degrading performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑