arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07337cs.LG
收藏

将知识从大型基础模型迁移到小型下游模型

Transferring Knowledge from Large Foundation Models to Small Downstream Models

New York University(纽约大学);AWS AI Labs(亚马逊云科技人工智能实验室)

Shikai Qiu, Boran Han, Danielle C. Maddix, Shuai Zhang, Yuyang Wang, Andrew Gordon Wilson

AI总结:

本文提出Adaptive Feature Transfer(AFT),通过自适应选择并迁移对下游任务最有用的预训练特征,实现低成本小模型对大型或多个基础模型知识的有效利用。

摘要:

我们如何将相关知识从规模不断增大的基础模型迁移到能够以低得多成本运行的小型、任务专用下游模型?使用预训练权重作为初始化的标准迁移学习只能迁移有限信息,并使我们受制于往往非常庞大的预训练架构。这一过程还阻碍了组合学习互补信息的多个预训练模型。为解决这些不足,我们提出自适应特征迁移(Adaptive Feature Transfer,AFT)。AFT不迁移权重,而是纯粹作用于特征,从而将预训练模型的选择与较小的下游模型解耦。AFT并非不加区分地压缩所有预训练特征,而是利用一种开销极小的简单正则化,自适应地迁移对执行下游任务最有用的预训练特征。在多个视觉、语言和多模态数据集上,与计算成本相近的替代方法相比,AFT取得了显著更优的下游性能。此外,即使下游模型缩小超过50倍,AFT仍能可靠地将预训练模型的提升转化为下游性能的提升,并且能够有效迁移多个预训练模型所学到的互补信息。

英文摘要:

How do we transfer the relevant knowledge from ever larger foundation models into small, task-specific downstream models that can run at much lower costs? Standard transfer learning using pre-trained weights as the initialization transfers limited information and commits us to often massive pre-trained architectures. This procedure also precludes combining multiple pre-trained models that learn complementary information. To address these shortcomings, we introduce Adaptive Feature Transfer (AFT). Instead of transferring weights, AFT operates purely on features, thereby decoupling the choice of the pre-trained model from the smaller downstream model. Rather than indiscriminately compressing all pre-trained features, AFT adaptively transfers pre-trained features that are most useful for performing the downstream task, using a simple regularization that adds minimal overhead. Across multiple vision, language, and multi-modal datasets, AFT achieves significantly better downstream performance compared to alternatives with a similar computational cost. Furthermore, AFT reliably translates improvement in pre-trained models into improvement in downstream performance, even if the downstream model is over $50\times$ smaller, and can effectively transfer complementary information learned by multiple pre-trained models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑