arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2101.02757cs.LG
收藏

基于权值注入的不同体系结构间的迁移学习

Transfer Learning Between Different Architectures Via Weights Injection

Poznan University of Technology(波兹南理工大学)

Maciej A. Czyzewski

摘要:

这项工作提出了一个天真的算法,不同的体系结构之间的参数传输与计算成本低的注入技术(不需要数据)。主要目标是从零开始加速神经网络的训练。本研究发现,从任何架构转移知识优于Kaiming和Xavier的初始化。总之,所提出的方法收敛速度更快,这使得它成为经典方法的替代品。该方法包括:1)匹配:预训练模型的层与目标模型; 2)注入:张量被转换成所需的形状。这项工作通过利用TLI(注入式迁移学习)评分,比较了当前SOTA架构(ImageNet)之间的相似性。

英文摘要:

This work presents a naive algorithm for parameter transfer between different architectures with a computationally cheap injection technique (which does not require data). The primary objective is to speed up the training of neural networks from scratch. It was found in this study that transferring knowledge from any architecture was superior to Kaiming and Xavier for initialization. In conclusion, the method presented is found to converge faster, which makes it a drop-in replacement for classical methods. The method involves: 1) matching: the layers of the pre-trained model with the targeted model; 2) injection: the tensor is transformed into a desired shape. This work provides a comparison of similarity between the current SOTA architectures (ImageNet), by utilising TLI (Transfer Learning by Injection) score.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑