arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.18153cs.LGcs.AI
收藏

基于扩散的神经网络权重生成

Diffusion-Based Neural Network Weights Generation

KAIST(韩国科学技术院);University of Freiburg(弗莱堡大学);DeepAuto;Meta AI

Bedionita Soro, Bruno Andreis, Hayeon Lee, Wonyong Jeong, Song Chong, Frank Hutter, Sung Ju Hwang

AI总结:

本文提出 D2NWG,利用潜在扩散与生成式超表征学习建模神经网络权重分布,从而依据目标数据集生成可泛化、可扩展至 LLM 的迁移学习权重。

摘要:

迁移学习因其能够加速收敛并提升新任务上的性能,近年来在深度学习研究中受到广泛关注。然而,其成功往往取决于源数据与目标数据之间的相似性,而且在大量数据集上进行训练成本高昂,导致人们盲目选择预训练模型,对其有效性认识有限。为应对这些挑战,我们提出 D2NWG,这是一种基于扩散的神经网络权重生成技术,以目标数据集为条件,高效生成用于迁移学习的高性能权重。我们的方法扩展了生成式超表征学习,将潜在扩散范式重塑用于神经网络权重生成,学习在各种数据集上预训练的模型的权重分布。这使得自动生成的权重能够在已见和未见任务上良好泛化,优于最先进的元学习方法和预训练模型。此外,我们的方法可扩展到大型语言模型(LLM)等大型架构,克服了当前参数生成技术依赖任务特定模型集合或访问原始训练数据的局限。通过对 LLM 的参数分布建模,D2NWG 无需额外微调或大量模型变体集合即可实现任务特定参数生成。大量实验表明,我们的方法持续提升了各种基础模型的性能,而不受其规模或复杂性影响,使其成为可扩展迁移学习的稳健解决方案。

英文摘要:

Transfer learning has gained significant attention in recent deep learning research due to its ability to accelerate convergence and enhance performance on new tasks. However, its success is often contingent on the similarity between source and target data, and training on numerous datasets can be costly, leading to blind selection of pretrained models with limited insight into their effectiveness. To address these challenges, we introduce D2NWG, a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning, conditioned on the target dataset. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation, learning the weight distributions of models pretrained on various datasets. This allows for automatic generation of weights that generalize well across both seen and unseen tasks, outperforming state-of-the-art meta-learning methods and pretrained models. Moreover, our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques that rely on task-specific model collections or access to original training data. By modeling the parameter distribution of LLMs, D2NWG enables task-specific parameter generation without requiring additional fine-tuning or large collections of model variants. Extensive experiments show that our method consistently enhances the performance of diverse base models, regardless of their size or complexity, positioning it as a robust solution for scalable transfer learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑