留在预训练盆地还是离开：关于迁移学习中集成的见解
To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning
AI总结：
本文研究从单个预训练模型微调得到的集成多样性受限问题，提出改进的StarSSE方法，通过更好地探索预训练盆地来提升集成质量和模型汤效果。
摘要：
迁移学习和集成是提升神经网络性能与鲁棒性的两种常用技术。由于预训练成本高昂，实践中常使用从单个预训练检查点微调得到的模型集成。这类模型最终会落入损失景观中的同一个盆地，我们称之为预训练盆地，因此其多样性有限。本文中，我们表明，从单个预训练检查点训练的集成可以通过更好地探索预训练盆地来改进；然而，离开该盆地会导致丧失迁移学习的收益，并使集成质量下降。基于对现有探索方法的分析，我们提出了一种针对迁移学习场景更有效的 Snapshot Ensembles（SSE）改进方案，即 StarSSE，它能够产生更强的集成和均匀的 model soups。
英文摘要：
Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. Due to the high cost of pre-training, ensembles of models fine-tuned from a single pre-trained checkpoint are often used in practice. Such models end up in the same basin of the loss landscape, which we call the pre-train basin, and thus have limited diversity. In this work, we show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, however, leaving the basin results in losing the benefits of transfer learning and in degradation of the ensemble quality. Based on the analysis of existing exploration methods, we propose a more effective modification of the Snapshot Ensembles (SSE) for transfer learning setup, StarSSE, which results in stronger ensembles and uniform model soups.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
