arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.02037cs.NEcond-mat.dis-nn
收藏

4Ward:一种用于高效训练任意复杂有向无环图的分层策略

4Ward: a Relayering Strategy for Efficient Training of Arbitrarily Complex Directed Acyclic Graphs

Tommaso Boccato, Matteo Ferrante, Andrea Duggento, Nicola Toschi

AI总结:

本文提出4Ward方法和Python库,利用图绘制分层算法从任意复杂有向无环图生成高效神经网络,通过并行化激活计算克服学习矩阵的顺序性并解决可扩展性问题,在Erdős-Rényi图实验中显著提升计算时间。

摘要:

得益于易于实现的特点,多层感知机(MLPs)在深度学习应用中已变得无处不在。MLP底层的图确实是多分图,即每一层神经元仅连接到相邻层的神经元。相比之下,个体突触水平的体内大脑连接组表明,生物神经网络具有无标度度分布或指数截断幂律强度分布的特征,暗示了利用进化衍生神经网络的潜在新途径。在本文中,我们提出了“4Ward”,这是一种方法和Python库,能够从任意复杂的有向无环图生成灵活高效的神经网络(NNs)。4Ward受图绘制领域的分层算法启发,以实现高效的前向传播,并在各种Erdős-Rényi图的计算实验中提供了显著的时间收益。4Ward不仅通过并行化激活计算克服了学习矩阵方法的顺序特性,还解决了当前最先进技术中遇到的可扩展性问题,并为设计者提供了自定义权重初始化和激活函数的自由。我们的算法可帮助任何寻求在微观尺度的神经网络设计框架中利用复杂拓扑结构的研究者。

英文摘要:

Thanks to their ease of implementation, multilayer perceptrons (MLPs) have become ubiquitous in deep learning applications. The graph underlying an MLP is indeed multipartite, i.e. each layer of neurons only connects to neurons belonging to the adjacent layer. In contrast, in vivo brain connectomes at the level of individual synapses suggest that biological neuronal networks are characterized by scale-free degree distributions or exponentially truncated power law strength distributions, hinting at potentially novel avenues for the exploitation of evolution-derived neuronal networks. In this paper, we present ``4Ward'', a method and Python library capable of generating flexible and efficient neural networks (NNs) from arbitrarily complex directed acyclic graphs. 4Ward is inspired by layering algorithms drawn from the graph drawing discipline to implement efficient forward passes, and provides significant time gains in computational experiments with various Erdős-Rényi graphs. 4Ward not only overcomes the sequential nature of the learning matrix method, by parallelizing the computation of activations, but also addresses the scalability issues encountered in the current state-of-the-art and provides the designer with freedom to customize weight initialization and activation functions. Our algorithm can be of aid for any investigator seeking to exploit complex topologies in a NN design framework at the microscale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑