融合微调模型以实现更好的预训练
Fusing finetuned models for better pretraining
摘要：
预训练的模型是训练的标准起点。这种方法始终优于使用随机初始化。然而，预先培训是一项代价高昂的工作，很少有人能够承担。在本文中，我们通过将多个现有的微调模型融合为一个，几乎不需要任何成本就可以创建更好的基础模型。具体来说，我们通过平均这些模型的权重进行融合。我们表明，融合模型的结果优于预训练的模型。我们还表明，融合往往比intertraining更好。我们发现，融合是较少依赖于目标任务。此外，权重衰减使训练间效果无效，但不会使融合效果无效。
英文摘要：
Pretrained models are the standard starting point for training. This approach consistently outperforms the use of a random initialization. However, pretraining is a costly endeavour that few can undertake. In this paper, we create better base models at hardly any cost, by fusing multiple existing fine tuned models into one. Specifically, we fuse by averaging the weights of these models. We show that the fused model results surpass the pretrained model ones. We also show that fusing is often better than intertraining. We find that fusing is less dependent on the target task. Furthermore, weight decay nullifies intertraining effects but not those of fusing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
