arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.10090cs.LG
收藏

权重空间学习综述:理解、表示与生成

A Survey of Weight Space Learning: Understanding, Representation, and Generation

Xiaolong Han, Zehong Wang, Bo Zhao, Binchi Zhang, Jundong Li, Damian Borth, Rose Yu, Haggai Maron, Yanfang Ye, Lu Yin, Ferrante Neri

AI总结:

本文综述了权重空间学习,探讨了权重空间的理解、表示与生成方法,及其在模型分析、转移和生成中的应用。

摘要:

神经网络权重通常被视为训练的最终产物,而大多数深度学习研究则聚焦于数据、特征和架构。然而,最近的进展表明,所有可能的权重值集合(权重空间)本身蕴含丰富的结构:预训练模型形成有序分布,表现出对称性,并可以被嵌入、比较甚至生成。理解此类结构对神经网络的分析和比较,以及跨模型的知识转移具有巨大影响,超越了单个训练实例。我们称之为权重空间学习(WSL)的研究方向,将神经权重视为有意义的分析和建模领域。本文提供了首个统一的WSL分类法。我们将现有方法分为三个核心维度:权重空间理解(WSU),研究权重的几何和对称性;权重空间表示(WSR),学习模型权重的嵌入;以及权重空间生成(WSG),通过超网络或生成模型合成新权重。我们进一步展示了这些发展如何使实际应用成为可能,包括模型检索、持续学习和联邦学习、神经架构搜索以及无数据重建。通过在统一框架下整合碎片化的进展,本文突显了权重空间作为可学习、结构化的领域,在模型分析、转移和权重生成方面的日益增长的影响。我们还发布了配套资源https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning。

英文摘要:

Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑