arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.07962cs.CL
收藏

重新审视参数有效的调优:我们真的到了吗?

Revisiting Parameter-Efficient Tuning: Are We Really There Yet?

Sun Yat-sen University(中山大学);University of Cambridge(剑桥大学);University of Glasgow(格拉斯哥大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, Shangsong Liang

摘要:

参数高效调优(PETuning)方法已被许多人视为使用预训练语言模型(PLM)的新范例。与全模型微调相比,通过调整一小部分参数,PETuning方法声称已经实现了与微调相当甚至更好的性能。在这项工作中,我们退一步,重新审视这些PETuning方法进行第一次全面调查的培训和评估他们。我们发现当前研究中存在问题的验证和测试实践,以及PETuning方法的不稳定性,导致了不可靠的结论。当在真正公平的评估协议下进行比较时,PETuning不能产生持续的竞争力,而微调仍然是中高资源环境中表现最好的方法。我们深入研究了不稳定性的原因,并观察到可训练参数和训练迭代的数量是两个主要因素:减少可训练参数和延长训练迭代可能会导致PETuning方法的更高稳定性。

英文摘要:

Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the new paradigm for using pretrained language models (PLMs). By tuning just a fraction amount of parameters comparing to full model finetuning, PETuning methods claim to have achieved performance on par with or even better than finetuning. In this work, we take a step back and re-examine these PETuning methods by conducting the first comprehensive investigation into the training and evaluation of them. We found the problematic validation and testing practice in current studies, when accompanied by the instability nature of PETuning methods, has led to unreliable conclusions. When being compared under a truly fair evaluation protocol, PETuning cannot yield consistently competitive performance while finetuning remains to be the best-performing method in medium- and high-resource settings. We delve deeper into the cause of the instability and observed that the number of trainable parameters and training iterations are two main factors: reducing trainable parameters and prolonging training iterations may lead to higher stability in PETuning methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑