arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.13435cs.LG
收藏

用于有效迁移学习的可扩展权重重新参数化

Scalable Weight Reparametrization for Efficient Transfer Learning

Qualcomm Technologies, Inc.(高通技术公司)

Byeonggeun Kim, Jun-Tae Lee, Seunghan yang, Simyung Chang

摘要:

本文提出了一种新的,高效的迁移学习方法,称为可扩展的权重重新参数化(SWR),是高效和有效的多个下游任务。高效的迁移学习涉及利用在较大数据集上训练的预训练模型,并将其重新用于下游任务,目的是最大限度地重用预训练模型。然而,以前的工作导致了更新参数和特定任务模块的增加,导致了更多的计算,特别是对于微小的模型。此外,没有实际考虑控制更新参数的数量。为了解决这些问题,我们建议学习一个策略网络,它可以决定在哪里重新参数化预训练模型,同时遵守更新参数数量的给定约束。策略网络仅在迁移学习过程中使用,而不是之后。因此,我们的方法在拟议的多语言关键字识别和标准基准(ImageNet to Sketch)中获得了最先进的性能,同时需要零额外计算和显着更少的额外参数。

英文摘要:

This paper proposes a novel, efficient transfer learning method, called Scalable Weight Reparametrization (SWR) that is efficient and effective for multiple downstream tasks. Efficient transfer learning involves utilizing a pre-trained model trained on a larger dataset and repurposing it for downstream tasks with the aim of maximizing the reuse of the pre-trained model. However, previous works have led to an increase in updated parameters and task-specific modules, resulting in more computations, especially for tiny models. Additionally, there has been no practical consideration for controlling the number of updated parameters. To address these issues, we suggest learning a policy network that can decide where to reparametrize the pre-trained model, while adhering to a given constraint for the number of updated parameters. The policy network is only used during the transfer learning process and not afterward. As a result, our approach attains state-of-the-art performance in a proposed multi-lingual keyword spotting and a standard benchmark, ImageNet-to-Sketch, while requiring zero additional computations and significantly fewer additional parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑