arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2207.10951cs.LG
收藏

用于预训练和迁移学习的超表示

Hyper-Representations for Pre-Training and Transfer Learning

University of St. Gallen(圣加仑大学);Samsung - SAIT AI Lab(三星综合技术院人工智能实验室);Universitat Politècnica de Catalunya(加泰罗尼亚理工大学)

Konstantin Schürholt, Boris Knyazev, Xavier Giró-i-Nieto, Damian Borth

摘要:

给定模型动物园的神经网络权重的学习表示是一个新兴且具有挑战性的领域,具有从模型检查到神经架构搜索或知识蒸馏的许多潜在应用。最近,一个在动物园模型上训练的自动编码器能够学习一个超表示,它捕获了动物园中模型的内在和外在属性。在这项工作中,我们扩展了生成使用的超表示,以作为预训练对新模型权重进行采样。我们提出了逐层损失归一化,我们证明这是生成高性能模型的关键,以及基于超表示经验密度的采样方法。使用我们的方法生成的模型是多样的,性能良好,能够超越传统的迁移学习基线。我们的研究结果表明,通过超表示从模型动物园到新模型的知识聚合的潜力,从而为新的研究方向铺平了道路。

英文摘要:

Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we extend hyper-representations for generative use to sample new model weights as pre-training. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and a sampling method based on the empirical density of hyper-representations. The models generated using our methods are diverse, performant and capable to outperform conventional baselines for transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑