arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.13673cs.CL
收藏

参数有效学习生成算法的评价

Evaluating Parameter Efficient Learning for Generation

The Hong Kong University of Science Technology(香港科技大学);NVIDIA(英伟达)

Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan J. Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro

摘要:

参数有效学习方法(PERM)最近获得了极大的关注,因为它们为预训练的语言模型(PLM)提供了一种有效的方法来适应下游任务。然而,这些结论大多来自于对整个训练集的域内评估。在本文中,我们从三个新的角度对PERM和微调进行了比较:(1)样本和模型大小对域内评估的影响,(2)对未知领域和新数据集的泛化,以及(3)世代的忠实性。我们的研究结果表明,对于域内设置,(a)存在样本大小的交叉点,在使用较少样本进行训练时,PERM的性能优于微调,(b)较大的PLM具有较大的交叉点。对于跨域和跨数据集的情况,我们表明(a)适配器(Houlsby等人,2019)在这里研究的所有PERM中表现最好,并且(b)如果任务数据集低于一定大小,则它优于微调。我们还比较了几代人的忠诚度,并表明PERM可以实现比微调更好的忠诚度得分,特别是对于小训练集,高达6%。最后,我们将适配器应用于MT-NLG 530 b(Smith等人,2022),并在Xsum上实现新的最先进的结果(Narayan等人,2018)的所有ROUGE评分(ROUGE-1 49.17,ROUGE-2 27.20,ROUGE-L 40.98)。

英文摘要:

Parameter efficient learning methods (PERMs) have recently gained significant attention as they provide an efficient way for pre-trained language models (PLMs) to adapt to a downstream task. However, these conclusions are mostly drawn from in-domain evaluations over the full training set. In this paper, we present comparisons between PERMs and finetuning from three new perspectives: (1) the effect of sample and model size to in-domain evaluations, (2) generalization to unseen domains and new datasets, and (3) the faithfulness of generations. Our results show that for in-domain settings (a) there is a cross point of sample size for which PERMs will perform better than finetuning when training with fewer samples, and (b) larger PLMs have larger cross points. For cross-domain and cross-dataset cases, we show that (a) Adapter (Houlsby et al., 2019) performs the best amongst all the PERMs studied here, and (b) it outperforms finetuning if the task dataset is below a certain size. We also compare the faithfulness of generations and show that PERMs can achieve better faithfulness score than finetuning, especially for small training set, by as much as 6%. Finally, we apply Adapter to MT-NLG 530b (Smith et al., 2022) and achieve new state-of-the-art results on Xsum (Narayan et al., 2018) for all ROUGE scores (ROUGE-1 49.17, ROUGE-2 27.20, ROUGE-L 40.98).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑