arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.01128cs.LGcs.AI
收藏

Mnemosyne:用Transformer学习训练Transformer

Mnemosyne: Learning to Train Transformers with Transformers

Google DeepMind(谷歌DeepMind);Columbia University(哥伦比亚大学);Google Research(谷歌研究院)

Deepali Jain, Krzysztof Marcin Choromanski, Avinava Dubey, Sumeet Singh, Vikas Sindhwani, Tingnan Zhang, Jie Tan

AI总结:

提出基于时空低秩隐式注意力Transformer的可学习优化器Mnemosyne,无需任务特定调参即可训练包括Transformer在内的网络,在准确率上匹配手工优化器,并扩展到大规模参数训练。

摘要:

在这项工作中,我们提出了一类新的可学习优化器,称为Mnemosyne。它基于新颖的时空低秩隐式注意力Transformer,能够学习训练整个神经网络架构,包括其他Transformer,而无需任何针对特定任务的优化器调参。我们证明Mnemosyne:(a)优于流行的LSTM优化器(即使采用了新的特征工程来缓解LSTM的灾难性遗忘问题);(b)能够在使用需要最少计算资源的简单元训练策略时成功训练Transformer;(c)在准确率上匹配经过精心调整超参数的手工设计的最先进优化器(通常产生性能最佳的模型)。此外,Mnemosyne提供了与其手工设计的一阶对应物相当的空间复杂度,使其能够扩展到训练更大规模的参数集。我们对Mnemosyne进行了广泛的实证评估:(a)微调各种规模的视觉Transformer(ViT),从中等规模架构到大型ViT-H(36层,16头);(b)预训练BERT模型;(c)对大型11B+ T5XXL模型进行软提示调优。我们通过全面的理论分析补充了我们的结果,该分析针对Mnemosyne使用的紧凑关联记忆,我们认为这从未被做过。

英文摘要:

In this work, we propose a new class of learnable optimizers, called \textit{Mnemosyne}. It is based on the novel spatio-temporal low-rank implicit attention Transformers that can learn to train entire neural network architectures, including other Transformers, without any task-specific optimizer tuning. We show that Mnemosyne: (a) outperforms popular LSTM optimizers (also with new feature engineering to mitigate catastrophic forgetting of LSTMs), (b) can successfully train Transformers while using simple meta-training strategies that require minimal computational resources, (c) matches accuracy-wise SOTA hand-designed optimizers with carefully tuned hyper-parameters (often producing top performing models). Furthermore, Mnemosyne provides space complexity comparable to that of its hand-designed first-order counterparts, which allows it to scale to training larger sets of parameters. We conduct an extensive empirical evaluation of Mnemosyne on: (a) fine-tuning a wide range of Vision Transformers (ViTs) from medium-size architectures to massive ViT-Hs (36 layers, 16 heads), (b) pre-training BERT models and (c) soft prompt-tuning large 11B+ T5XXL models. We complement our results with a comprehensive theoretical analysis of the compact associative memory used by Mnemosyne which we believe was never done before.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑