arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.09997cs.LG
收藏

迈向可扩展且通用的权重空间学习

Towards Scalable and Versatile Weight Space Learning

University of St.Gallen(圣加仑大学);International Computer Science Institute(国际计算机科学研究所);University of California at Berkeley(加州大学伯克利分校);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)

Konstantin Schürholt, Michael W. Mahoney, Damian Borth

AI总结:

本文提出 SANE 方法,通过顺序处理权重子集实现可扩展且任务无关的神经网络权重空间学习,在多个基准上达到或超过最先进性能,并能生成未见过的模型。

摘要:

学习训练良好的神经网络模型的表征有望帮助理解这些模型的内部工作机制。然而,以往的工作要么在处理较大网络时面临局限性,要么仅针对判别式或生成式任务。本文提出了用于权重空间学习的 SANE 方法。SANE 通过学习神经网络的任务无关表征,克服了以往的局限性,能够扩展到不同架构的更大模型,并展现出超越单一任务的能力。我们的方法将超表征的思想扩展为对神经网络权重的子集进行顺序处理,从而允许将更大的神经网络作为一组 token 嵌入到学习到的表征空间中。SANE 从逐层嵌入中揭示全局模型信息,并且能够顺序生成未见过的神经网络模型,这是以往超表征学习方法无法实现的。大量实证评估表明,SANE 在多个权重表征学习基准上达到或超过了最先进的性能,尤其是在新任务初始化和更大的 ResNet 架构上。

英文摘要:

Learning representations of well-trained neural network models holds the promise to provide an understanding of the inner workings of those models. However, previous work has either faced limitations when processing larger networks or was task-specific to either discriminative or generative tasks. This paper introduces the SANE approach to weight-space learning. SANE overcomes previous limitations by learning task-agnostic representations of neural networks that are scalable to larger models of varying architectures and that show capabilities beyond a single task. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights, thus allowing one to embed larger neural networks as a set of tokens into the learned representation space. SANE reveals global model information from layer-wise embeddings, and it can sequentially generate unseen neural network models, which was unattainable with previous hyper-representation learning methods. Extensive empirical evaluation demonstrates that SANE matches or exceeds state-of-the-art performance on several weight representation learning benchmarks, particularly in initialization for new tasks and larger ResNet architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑