arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.14224cs.NE
收藏

面向神经进化的拼接:在不破坏深度神经网络的前提下重组它们

Stitching for Neuroevolution: Recombining Deep Neural Networks without Breaking Them

Arthur Guijt, Dirk Thierens, Tanja Alderliesten, Peter A. N. Bosman

AI总结:

提出一种基于拼接的神经进化方法,通过匹配父网络连接性并在交叉点引入新层来重组预训练深度神经网络,从而高效发现性能与计算成本权衡更优的新模型。

摘要:

传统的神经进化方法通常从零开始。当目标为现代深度神经网络时,这在计算和数据需求方面变得极其昂贵。使用热启动会非常有利,例如使用之前训练过的网络,这些网络可能来自不同来源。此外,这还能利用迁移学习的优势(特别是大幅减少训练工作量)。然而,重组已训练的网络并非易事,因为架构和特征表示通常不同。因此,直接交换层往往会导致性能崩溃。我们通过基于连接性匹配父网络的层来克服这一问题,从而识别潜在的交叉点。为了纠正这些层之间不同的特征表示,我们采用了拼接技术,通过在交叉点引入新层来合并网络。为了训练合并后的网络,只需考虑拼接层。随后,通过选择使用或不使用哪些拼接层来选取子网络,即可创建新网络。评估其性能非常高效,因为只需在数据上进行评估即可。我们通过实验证明,该方法能够找到在性能和计算成本之间呈现新权衡的网络,其中一些甚至超越了原始网络。

英文摘要:

Traditional approaches to neuroevolution often start from scratch. This becomes prohibitively expensive in terms of computational and data requirements when targeting modern, deep neural networks. Using a warm start could be highly advantageous, e.g., using previously trained networks, potentially from different sources. This moreover enables leveraging the benefits of transfer learning (in particular vastly reduced training effort). However, recombining trained networks is non-trivial because architectures and feature representations typically differ. Consequently, a straightforward exchange of layers tends to lead to a performance breakdown. We overcome this by matching the layers of parent networks based on their connectivity, identifying potential crossover points. To correct for differing feature representations between these layers we employ stitching, which merges the networks by introducing new layers at crossover points. To train the merged network, only stitching layers need to be considered. New networks can then be created by selecting a subnetwork by choosing which stitching layers to (not) use. Assessing their performance is efficient as only their evaluation on data is required. We experimentally show that our approach enables finding networks that represent novel trade-offs between performance and computational cost, with some even dominating the original networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑