智能学习率分布以减少Transformer中的灾难性遗忘
Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers
AI总结：
本文针对Transformer微调中的灾难性遗忘问题，通过超参数优化寻找并组合优于平坦学习率的学习率分布，并在GLUE基准上验证其有效性。
摘要：
在自然语言处理中，在大规模文本语料库上预训练语言模型是一种常见做法。随后对这些模型进行微调，以在多种任务上取得最佳结果。在本文中，我们研究了Transformer神经网络中的灾难性遗忘问题，并对在此背景下对整个网络使用平坦学习率进行微调的常见做法提出质疑。我们执行超参数优化过程，以寻找优于平坦学习率的学习率分布。我们将由此找到的学习率分布进行组合，并表明它们在灾难性遗忘问题上能够泛化到更好的性能。我们使用来自GLUE数据集的各种NLP基准验证了这些学习率分布。
英文摘要：
Pretraining language models on large text corpora is a common practice in natural language processing. Fine-tuning of these models is then performed to achieve the best results on a variety of tasks. In this paper, we investigate the problem of catastrophic forgetting in transformer neural networks and question the common practice of fine-tuning with a flat learning rate for the entire network in this context. We perform a hyperparameter optimization process to find learning rate distributions that are better than a flat learning rate. We combine the learning rate distributions thus found and show that they generalize to better performance with respect to the problem of catastrophic forgetting. We validate these learning rate distributions with a variety of NLP benchmarks from the GLUE dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
