不要把你的学习率扫到地毯下：仔细看看预训练的Transformers的跨模态迁移
Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers
摘要：
在文本语料库上对大规模Transformer模型进行自我监督预训练，然后进行微调，在许多自然语言处理任务上取得了最先进的水平。最近，Lu等人（2021，arXiv：2103.05247）声称，在一组转移任务中，冻结的预训练Transformers（FPT）匹配或优于从头开始的训练以及未冻结（微调）的预训练Transformers。在我们的工作中，我们发现这个结果实际上是没有调优学习率的结果。在仔细重新设计经验设置后，我们发现，当适当调整学习率时，预训练的Transformers在我们所有的任务中都优于或匹配从头开始的训练，但只有当整个模型被微调时。因此，虽然从预训练的语言模型转移到其他模态确实为未来的工作提供了令人兴奋的可能性，但正确调整超参数对于获得强大的发现非常重要。
英文摘要：
Self-supervised pre-training of large-scale transformer models on text corpora followed by finetuning has achieved state-of-the-art on a number of natural language processing tasks. Recently, Lu et al. (2021, arXiv:2103.05247) claimed that frozen pretrained transformers (FPTs) match or outperform training from scratch as well as unfrozen (fine-tuned) pretrained transformers in a set of transfer tasks to other modalities. In our work, we find that this result is, in fact, an artifact of not tuning the learning rates. After carefully redesigning the empirical setup, we find that when tuning learning rates properly, pretrained transformers do outperform or match training from scratch in all of our tasks, but only as long as the entire model is finetuned. Thus, while transfer from pretrained language models to other modalities does indeed provide gains and hints at exciting possibilities for future work, properly tuning hyperparameters is important for arriving at robust findings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
