arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.14764cs.LG
收藏

Model Zoos:一个包含不同神经网络模型种群的数据集

Model Zoos: A Dataset of Diverse Populations of Neural Network Models

University of St.Gallen(圣加仑大学);Samsung - SAIT AI Lab(三星-三星先进技术研究院人工智能实验室);Institut de Robòtica i Informàtica Industrial(工业机器人与信息技术研究所);Universitat Politècnica de Catalunya(加泰罗尼亚理工大学)

Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i-Nieto, Damian Borth

摘要:

在过去的几年里,神经网络(NN)已经从实验室环境发展到许多现实世界问题的最先进的。结果表明,NN模型(即,它们的权重和偏置)在训练期间在权重空间中的唯一轨迹上演变。接下来,这样的神经网络模型的群体(称为模型动物园)将在权重空间中形成结构。我们认为,这些结构的几何形状、曲率和光滑度包含了关于训练状态的信息,可以揭示单个模型的潜在属性。有了这样的模型动物园,人们可以研究新的方法(i)模型分析,(ii)发现未知的学习动态,(iii)学习这些种群的丰富表示,或(iv)利用模型动物园对NN权重和偏差进行生成建模。不幸的是,缺乏标准化的模型动物园和可用的基准显着增加了对NN种群的进一步研究的摩擦。通过这项工作,我们发布了一个新的模型动物园数据集,其中包含系统生成的和不同的NN模型种群,以供进一步研究。总的来说,所提出的模型动物园数据集基于8个图像数据集,由27个模型动物园组成,这些模型动物园使用不同的超参数组合进行训练,并且包括50 '360个唯一的NN模型以及它们的稀疏化双胞胎,从而产生超过3'844' 360个收集的模型状态。此外,对于模型动物园数据,我们提供了对动物园的深入分析,并为多个下游任务提供了基准。该数据集可在www.modelzoos.cc上找到。

英文摘要:

In the last years, neural networks (NN) have evolved from laboratory environments to the state-of-the-art for many real-world problems. It was shown that NN models (i.e., their weights and biases) evolve on unique trajectories in weight space during training. Following, a population of such neural network models (referred to as model zoo) would form structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can reveal latent properties of individual models. With such model zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of NN weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of NNs. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of NN models for further research. In total the proposed model zoo dataset is based on eight image datasets, consists of 27 model zoos trained with varying hyperparameter combinations and includes 50'360 unique NN models as well as their sparsified twins, resulting in over 3'844'360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks. The dataset can be found at www.modelzoos.cc.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑