Towards Scalable and Versatile Weight Space Learning
University of St.Gallen(圣加仑大学);International Computer Science Institute(国际计算机科学研究所);University of California at Berkeley(加州大学伯克利分校);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)
Konstantin Schürholt, Michael W. Mahoney, Damian Borth
Learning representations of well-trained neural network models holds the promise to provide an understanding of the inner workings of those models. However, previous work has either faced limitations when processing larger networks or was task-specific to either discriminative or generative tasks. This paper introduces the SANE approach to weight-space learning. SANE overcomes previous limitations by learning task-agnostic representations of neural networks that are scalable to larger models of varying architectures and that show capabilities beyond a single task. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights, thus allowing one to embed larger neural networks as a set of tokens into the learned representation space. SANE reveals global model information from layer-wise embeddings, and it can sequentially generate unseen neural network models, which was unattainable with previous hyper-representation learning methods. Extensive empirical evaluation demonstrates that SANE matches or exceeds state-of-the-art performance on several weight representation learning benchmarks, particularly in initialization for new tasks and larger ResNet architectures.
Accelerated Dynamic Importance Weighting with Versatile Divergence-Minimizing Estimators
The Institute of Statistical Mathematics(统计数学研究所);University of Bristol(布里斯托大学);RIKEN Center for Advanced Intelligence Project(RIKEN高级智能项目中心);The University of Tokyo(东京大学)
Scalable Weight Reparametrization for Efficient Transfer Learning
Qualcomm Technologies, Inc.(高通技术公司)
本文提出了一种新的,高效的迁移学习方法,称为可扩展的权重重新参数化(SWR),是高效和有效的多个下游任务。高效的迁移学习涉及利用在较大数据集上训练的预训练模型,并将其重新用于下游任务,目的是最大限度地重用预训练模型。然而,以前的工作导致了更新参数和特定任务模块的增加,导致了更多的计算,特别是对于微小的模型。此外,没有实际考虑控制更新参数的数量。为了解决这些问题,我们建议学习一个策略网络,它可以决定在哪里重新参数化预训练模型,同时遵守更新参数数量的给定约束。策略网络仅在迁移学习过程中使用,而不是之后。因此,我们的方法在拟议的多语言关键字识别和标准基准(ImageNet to Sketch)中获得了最先进的性能,同时需要零额外计算和显着更少的额外参数。
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);School of Data Science, Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Shenzhen Research Institute of Big Data(深圳大数据研究院);Tencent(腾讯)
Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization
School of Computer Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其跨学科应用重点实验室(东南大学),教育部,中国)
ZoRRO: A Zero-Weight Personalized Recommender System for Scalable News Recommendation
Technical University of Denmark(丹麦技术大学);ZOZO Research(ZOZO研究公司);University of California San Diego(加州大学圣地亚哥分校);Pioneer Centre for Artificial Intelligence(先锋人工智能中心)
Self-Supervised Weight Templates for Scalable Vision Model Initialization
School of Computer Science and Engineering, Southeast University, Nanjing, China(计算机科学与工程学院,东南大学,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用重点实验室(东南大学),教育部,中国)
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);University of British Columbia(不列颠哥伦比亚大学);University of Oxford(牛津大学);University of Edinburgh(爱丁堡大学)
Spin-Weighted Spherical Harmonics Enable Complete and Scalable $\mathrm{E}(3)$-Equivariant Networks
Department of Computer Science and Engineering, Texas A&M University(德克萨斯农工大学计算机科学与工程系);Lambda, Inc.(Lambda公司);Department of Materials Science and Engineering, Texas A&M University(德克萨斯农工大学材料科学与工程系);Department of Electrical and Computer Engineering, Texas A&M University(德克萨斯农工大学电气与计算机工程系);Department of Physics and Astronomy, Texas A&M University(德克萨斯农工大学物理与天文学系);Department of Mechanical Engineering, Texas A&M University(德克萨斯农工大学机械工程系)
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
The Hong Kong Polytechnic University, PolyU(香港理工大学,PolyU);Hong Kong Polytechnic University Daya Bay Technology(香港理工大学达亚拜技术);Innovation Research Institute(创新研究院)