arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.09204cs.CL
收藏

预训练语言模型微调的超参数优化实证研究

An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models

Stevens Institute of Technology(史蒂文斯理工学院);Microsoft Research(微软研究院)

Xueqing Liu, Chi Wang

摘要:

微调预训练的语言模型的性能在很大程度上取决于超参数配置。在本文中,我们研究了现代超参数优化方法(HPO)对预训练语言模型进行微调的性能。首先,我们研究并报告了三种HPO算法在GLUE数据集上对两种最先进的语言模型进行微调的性能。我们发现,使用相同的时间预算,HPO往往无法优于网格搜索由于两个原因:时间预算不足和过拟合。我们提出了两个一般性的战略和实验程序,系统地解决HPO的故障情况。通过应用该过程,我们观察到HPO可以在搜索空间和时间预算中使用更适当的设置来成功;然而,在某些情况下仍然存在过拟合。最后,我们对未来的工作提出了建议。我们的实现可以在https://github.com/microsoft/FLAML/tree/main/flaml/nlp/中找到。

英文摘要:

The performance of fine-tuning pre-trained language models largely depends on the hyperparameter configuration. In this paper, we investigate the performance of modern hyperparameter optimization methods (HPO) on fine-tuning pre-trained language models. First, we study and report three HPO algorithms' performances on fine-tuning two state-of-the-art language models on the GLUE dataset. We find that using the same time budget, HPO often fails to outperform grid search due to two reasons: insufficient time budget and overfitting. We propose two general strategies and an experimental procedure to systematically troubleshoot HPO's failure cases. By applying the procedure, we observe that HPO can succeed with more appropriate settings in the search space and time budget; however, in certain cases overfitting remains. Finally, we make suggestions for future work. Our implementation can be found in https://github.com/microsoft/FLAML/tree/main/flaml/nlp/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑